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ABSTRACT 

This  document  provides  a  generic,  efficient  and  cost-effective  method  of  comparing  system 
instantiations.  It  gives  a  unique  view  of  a  complex  system  by  considering  how  well  the  system 
supports  the  overarching  aim.  The  method  proposed  was  developed  from  a  requirement  to 
undertake  evaluations  of  complex  military  systems,  in  particular  those  associated  with  command 
and  control.  To  illustrate  the  generic  nature  of  the  method,  it  is  introduced  using  three  very 
different  systems.  A  military  headquarters  is  then  used  as  a  more  comprehensive  example  of  how 
the  method  can  be  applied. 
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System  Instantiation  Comparison  Method:  A 
Technique  for  Comparing  Military  Headquarters 


Executive  Summary 

Investigating  complex  systems,  such  as  a  military  headquarters,  poses  a  challenge,  not 
only  in  terms  of  generically  defining  the  system,  but  also  in  providing  a  flexible  and  cost- 
effective  way  of  comparing  similar  types  of  systems.  Introducing  new  technologies  adds 
further  complexity  to  this  problem.  Many  methods  and  techniques  have  been  developed 
to  evaluate  systems.  However  these  are  process  orientated  and  output  driven,  focusing  on 
the  quality  of  the  outputs  and  processes  of  the  system.  These  processes  can,  and  usually 
do,  change  from  one  system  of  a  similar  type  to  another,  they  are  not  always  generic  in 
nature  and  may  not  even  add  to  the  overall  goal  of  the  system.  Hence,  using  these 
methods  to  make  comparisons  between  similar  types  of  systems  is  difficult.  A  new  way  of 
viewing  a  complex  system  and  providing  techniques  for  the  comparison  of  system 
instantiations  is  required.  This  document  introduces  such  a  system  instantiation 
comparison  technique. 

In  systems  methodologies,  the  system  instantiation  comparison  method  is  aimed  at  use  in 
analysing  complex  systems;  in  particular,  those  systems  where  there  are  multiple  decision 
makers  who  do  not  all  have  the  same  goal  or  understanding  of  the  current  situation.  What 
is  unique  about  the  method  is  the  view  that  it  takes  of  the  system.  Here,  a  complex  system 
is  defined  in  terms  of  how  'well'  it  allows  you  to  achieve  its  operational  objectives.  This 
places  no  judgement  on  the  quality  of  the  outputs  or  the  processes  of  the  system.  The 
method  described  defines  a  generic  system  in  terms  of  its  variable  and  static  properties 
providing  a  foundation  for  comparative  evaluations.  It  defines  a  complex  system  in  simple 
terms  and  provides  a  way  of  quantifying  the  different  options  for  the  same  type  of  system, 
thus  allowing  for  a  direct  comparison  between  instantiations.  It  also  provides  a  scaling 
ability  for  the  level  of  detail  in  the  application  of  the  method;  if  required,  more  detail  can 
be  included  in  subsequent  studies.  Due  to  the  way  the  system  is  defined  by  the  method, 
comparisons  of  instantiations  are  considered  by  the  authors  to  be  efficient  and  cost- 
effective. 

The  method  proposed  was  developed  from  a  requirement  to  undertake  evaluations  of  a 
complex  military  system,  in  particular  evaluations  associated  with  military  command  and 
control  and  the  introduction  of  new  technology.  The  method  is  first  introduced  using  three 
very  different  systems  to  illustrate  its  generic  nature  and  flexibility.  A  military 
headquarters  is  used  as  a  more  comprehensive  example  of  how  the  method  can  be 
applied.  Generic  characteristics  associated  with  the  information  and  the  functions  used  by 
a  military  headquarters,  as  defined  by  the  authors,  are  used  as  the  basis  to  represent  the 
system.  System  measures  associated  with  different  instantiations  were  obtained  and  rated. 


Authors 


Leanne  Mary  Laidlaw  Rees 

Land  Operations  Division 

During  Leanne's  23-year  career  at  DSTO  she  has  worked  mainly  in 
the  areas  of  operations  research  and  analysis,  where  she  has  been 
responsible  for  the  management  and  scientific  conduct  in  a  diverse 
range  of  areas,  including  underwater  warfare,  electronic  warfare, 
intelligence  applications,  conduct  of  military  operations,  air  defence, 
battlefield  command  support  systems,  unmanned  aerial  vehicles, 
radar,  communications,  headquarters  operations,  air,  land  and 
maritime  operations,  synthetic  environments,  surveillance  and 
reconnaissance,  simulation  and  war  gaming  and  conducting  studies 
into  the  introduction  of  new  technologies  into  military  environments. 
She  deployed  operationally  to  East  Timor  in  2001  as  an  analyst. 
Leanne  has  a  Bachelor  of  Applied  Science  in  Applied  Physics  from  the 
South  Australian  Institute  of  Technology  and  is  currently  employed 
as  a  Senior  Analyst  in  DSTO's  Land  Operations  Division,  focusing 
on  Command,  Control,  Communications  and  Intelligence;  Army 
Aviation;  evaluation  of  complex  systems;  and  operational 
effectiveness.  Her  work  has  focused  on  providing  high  quality 
research  and  its  application  to  the  Australian  Defence  Porce. 


Fred  D  J  Bowden 

Land  Operations  Division 

Pred  Bowden  completed  his  Bachelor  of  Science  at  Murdoch 
University  majoring  in  Mathematics  and  Physics.  He  joined  DSTO 
in  1990.  While  working  for  DSTO  Pred  completed  a  First  Class 
Honours  degree  and  PhD  in  Applied  Mathematics  at  the  University 
of  Adelaide.  The  focus  of  his  work  in  the  90s  was  military  Command 
and  Control  systems.  He  spent  late  2001  to  early  2003  on  a  Defence 
Science  Fellowship  with  the  RAND  organisation  in  Washington  DC 
working  on  NCW  and  C2  analysis.  Since  returning  from  RAND  he 
has  led  the  analysis  conducted  by  DSTO  for  the  Army  Experimental 
Framework. 


Contents 


1.  INTRODUCTION . 1 

2.  AN  OVERVIEW  OF  PRIMARY  EXISTING  METHODS . 1 

3.  THE  SYSTEM  INSTANTIATION  COMPARISON  METHOD . 5 

3.1  The  System  Model . 6 

3.1.1  System's  Operational  Requirement . 7 

3.1.2  Critical  Component  and  Critical  Characteristics . 8 

3.1.3  System  Functions . 9 

3.1.4  System  Enablers . 10 

3.1.5  Summary  of  Components . 12 

3.2  Using  the  Method  for  Evaluation . 18 

3.3  Getting  One  Measure  of  the  System . 19 

3.4  Relationship  of  Analysis  to  Context . 21 

4.  THE  MILITARY  HEADQUARTERS  MODEL . 22 

4.1  Defining  the  SIC  Method  Components  as  Applied  to  a  Military  Headquarters .  23 

4.2  Application  of  the  Method  to  the  Evaluation  of  a  Military  Headquarters . 25 

4.2.1  Ranking  the  Overall  System  Contexts . 29 

4.2.2  System  Cost . 31 

4.2.3  Calculation  of  Criticality  of  Information  Characteristics . 31 

5.  DISCUSSION  AND  CONCLUSIONS . 38 

6.  ACKNOWLEDGMENTS . 40 

7.  REFERENCES . 40 


Figures 


Figure  1:  The  System  Representation . 7 

Figure  2:  SIC  Method:  System  Definition  for  the  Example  of  a  Biological  System . 14 

Figure  3:  SIC  Method:  System  Definition  for  the  Example  of  an  Army  Tank  Procurement 

. 15 

Figure  4:  SIC  Method:  System  Definition  for  the  Example  of  an  Organisation . 16 

Figure  5:  Example  of  a  Military  Headquarters  Representation . 24 

Figure  6:  Ratio  for  a  Circular  Optimal  Boundary . 32 

Figure  7:  Ratio  for  a  Hyperbola  Optimal  Boundary . 34 

Figure  8:  Ratio  for  a  Linear  Optimal  Boundary . 35 

Figure  9:  Plot  of  Criticality  Values  Versus  Information  Characteristics . 38 


Tables 

Table  1:  Examples  of  the  components  of  The  System  Instantiation  Comparison  Method 


. 13 

Table  2:  Summary  of  Tools  for  System  Representation . 17 

Table  3:  System  Measure  Determination . 20 

Table  4:  Evaluation  For  The  First  Instantiation . 26 

Table  5:  Evaluation  for  the  Second  Instantiation . 27 

Table  6:  Evaluation  for  the  Third  Instantiation . 28 

Table  7:  Comparison  of  System  Instantiations . 29 

Table  8:  System  Instantiation  Measures  In  Relation  To  Context . 30 

Table  9:  Instantiation  Ranking  in  Relation  to  Maximum  and  Minimum  Instantiations30 
Table  10:  Perishability,  Frequency  and  Importance  for  Information  Characteristics...  37 
Table  11:  Criticality  Value  For  Each  Information  Category . 37 


ii 


DSTO-RR-0322 


1.  Introduction 


Complex  systems  by  their  very  nature  are  difficult  to  evaluate.  The  problems  lie  in 
defining  the  system  in  such  a  way  as  to  provide  a  flexible  and  cost-effective  comparison  of 
similar  systems.  A  new  way  of  viewing  a  complex  system  and  providing  techniques  for 
comparing  system  instantiations1  is  required.  This  paper  introduces  the  System 
Instantiation  Comparison  (SIC)  method,  intended  to  meet  the  shortfall  in  the  currently 
available  techniques. 

The  method  was  developed  to  meet  a  requirement  to  provide  a  simple,  flexible  and  cost- 
effective  approach  with  which  to  compare  complex  military  systems,  such  as  command 
and  control,  so  that  changes,  such  as  the  introduction  of  new  technologies,  in  these 
systems  can  be  quantified.  However,  the  method  can  also  be  applied  to  the  examination  of 
a  broad  class  of  systems  that  are  not  necessarily  military,  for  example  biological  systems.  It 
should  be  noted  that  the  method  has  been  designed  to  facilitate  the  comparison  of 
instantiations  of  a  system,  not  to  give  an  absolute  value  of  the  system.  That  is,  it  will  not 
give  the  analyst  a  quantitative  outcome  such  as  This  system  is  10%  effective'. 

First,  a  review  of  existing  techniques,  highlighting  deficiencies  for  the  analysis  techniques 
for  complex  systems  is  presented.  This  clearly  demonstrates  the  need  for  a  new  approach 
to  viewing  and  evaluating  complex  systems.  Next  the  SIC  method  is  introduced  in  a 
general  context  using  several  examples  of  complex  systems  to  illustrate  the  concepts.  This 
is  followed  by  a  more  in-depth  application  of  the  method  to  the  analysis  of  a  military 
headquarters.  Finally,  some  concluding  remarks  and  comments  on  future  directions  of  this 
research  are  outlined,  including  an  overview  of  where  it  has  been  applied  in  practice. 


2.  An  Overview  of  Primary  Existing  Methods 

There  are  many  different  system  analysis  approaches  described  in  the  literature.  One  way 
of  helping  to  determine  the  most  appropriate  technique  for  a  given  problem  is  to  use  a 
problem  classification  technique  as  defined  in  Flood  and  Jackson  (1991).  In  this 
classification,  the  state  space  of  problems  is  defined  as  a  2-D  space.  One  axis  is  labelled 
with  simple  and  complex  and  the  other  with  unitary,  pluralist  and  coercive2.  Thus,  using 
the  guidance  given  in  Flood  and  Jackson  (1991),  a  problem  can  be  classified  to  a  given 
section  of  problem  space  and  an  appropriate  analysis  method  applied.  The  SIC  method  is 
aimed  at  use  with  complex  pluralist  systems,  although  it  is  also  applicable  to  complex 
unitary  and,  in  a  more  limited  fashion,  to  complex  coercive  systems. 

Prior  to  the  development  of  the  SIC  method,  many  other  techniques  were  investigated  by 
the  authors.  None  of  these  were  considered  to  fully  satisfy  the  analysis  needs.  The 


1  An  instantiation  of  a  system  is  a  particular  variant  of  the  system. 

2  Definitions  for  each  of  these  can  be  found  in  Flood  and  Jackson  (1991). 
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following  is  a  brief  outline  of  some  of  these  approaches,  suggesting  the  relative  merits  and 
weaknesses  of  these  techniques  as  applied  to  the  comparison  of  complex  system 
instantiations. 

The  soft  systems  approach  is  a  technique  that  is  growing  in  popularity  and  provides  a 
descriptive  platform  on  which  to  base  the  investigation  of  systems  (Checkland  1993;  US 
DoD  2000a;  US  DoD  2000b;  Finegan  2000;  Lane  and  Galvin  1999;  and  Soft  Systems  2000).  A 
pictorial  representation  of  the  system  through  the  use  of  conceptual  models  provides  the 
analyst  with  an  impression  of  the  prevailing  trends  within  the  organisation.  The  use  of 
rich  pictures  provides  descriptions  of  the  system's  tasks  and  how  it  performs  them. 
However,  this  method  lacks  the  ability  to  easily  provide  a  quantitative  measure  on  which 
to  conduct  investigations  due  to  its  descriptive  nature. 

Knowledge-based  system  development  (ARTI 2000)  builds  a  number  of  separate  models 
that  capture  salient  features  of  the  system  and  environment,  including:  organisational; 
task;  agent;  communication;  and  expert  and  design  models.  It  is  process  based  and  does 
not  lend  itself  to  providing  a  complete  view  of  the  system;  rather  it  provides  many  views 
of  the  system.  This  is  good  in  that  it  gives  insight  into  aspects  of  the  system,  but  it  does  not 
give  a  good  idea  of  changes  to  the  system  as  a  whole. 

Quality  Function  Deployment  is  a  conceptual  map  that  provides  the  means  for  cross¬ 
functional  planning  and  communications  (Jagdev,  Bradley  and  Molley  1997).  It  is  a 
method  for  transforming  customer  wants  and  needs  into  quantitative,  engineering  terms. 
This  technique  focuses  on  the  characteristics  of  an  organisation's  products  and  services. 
These  characteristics  are  assessed  and  prioritised  using  the  customer's  or  end  user's  point 
of  view.  The  method  is  quantitative  in  nature,  it  does  provide  measuring  parameters  but 
does  not  provide  a  method  for  putting  them  into  a  single  number  to  represent  the  system, 
limiting  its  ability  to  be  used  to  compare  system  instantiations. 

The  structured  systems  analysis  and  design  methodology  is  a  typical  structured  method 
used  in  the  analysis  and  design  stages  of  systems  development,  but  it  is  not  well  suited  to 
the  construction,  testing  and  implementation  stages  (Uni.  of  Glamorgan  2000  and 
Universitat  Bremen  2000c).  This  method  seems  best  directed  towards  software 
development  or  information  systems  engineering.  The  technique  adopts  a  prescriptive 
approach  to  information  systems  development  in  that  it  specifies  in  advance  the  modules, 
stages  and  tasks  which  have  to  be  carried  out,  the  deliverables  to  be  produced  and  the 
techniques  used  to  produce  these  deliverables.  The  technique  adopts  the  waterfall  model 
(2000)  of  systems  development,  which  uses  a  systematic  sequential  approach,  where  each 
phase  has  to  be  completed  and  signed  off  before  the  next  can  begin.  The  application  of 
structured  systems  analysis  and  design  methods  are  aimed  primarily  at  the  development 
of  information  systems  on  the  basis  of  data  base  systems  and  less  at  the  development  of 
real  time  oriented  software.  The  method  concentrates  on  data  flows  and  data  models, 
which  suggests  that  it  is  best  suited  for  the  investigation  of  the  processes  carried  out  by  a 
system. 

Functional  decomposition,  as  defined  in  Universitat  Bremen  (2000c),  is  more  a  philosophy 
of  how  to  break  down  a  system.  The  approach  can  be  best  summarised  by  the  following 
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quote  from  Universitat  Bremen  (2000c):  'the  objective  is  to  decompose  a  system  step  by 
step,  beginning  with  the  main  function  of  a  system  and  continuing  with  the  interim  levels 
down  to  the  level  of  elementary  functions.  On  each  level,  abstractions  are  made  from  each 
corresponding  lower  level.  All  the  sub  functions  together  form  completely  the 
decomposed  function  (functional  hierarchy)'.  This  technique  appears  to  be  used  in  concert 
with  other  methods  to  produce  a  list  of  hierarchical  elements  rather  than  an  evaluation 
technique.  It  provides  components  under  which  to  view  the  system. 

Data  flow  modelling,  Universitat  Bremen  (2000a),  is  used  to  define  the  functional  structure 
of  a  system  by  means  of  the  combined  consideration  of  functions  and  data.  The  data  flows 
are  the  interfaces  between  the  functions.  The  data  flow  modelling  abstracts  from  the 
physical  facts  of  a  projected  system.  It  takes  a  top  down  approach,  where  more  and  more 
detailed  levels  of  the  future  systems  are  specified.  This  is  based  on  the  context  diagram, 
which  only  represents  the  data  flows  of  the  systems  from  and  to  its  environment.  When 
refining  the  data  flow  model,  the  functions  are  identified  in  the  functional  hierarchy 
(functional  decomposition)  and  refined  by  means  of  the  data  flow  diagram  of  the 
corresponding  levels.  The  data  flow  diagram  of  a  certain  hierarchical  level  can  be  made  to 
represent  the  incorporation  of  processes  that  are  connected  via  data  flows.  A  refinement  of 
the  data  flow  diagram  is  always  realised  in  balance  with  the  corresponding  refinement  of 
the  functional  hierarchy.  This  technique  focuses  on  describing  the  processes  used  in  a 
system  and  provides  only  limited  evaluation  as  it  is  only  concerned  with  the  suitability  of 
the  data  constructs. 

Activity  modelling  develops  an  accurate  description  of  the  activities  performed  by  the 
system.  These  models  seek  to  discover  'what'  needs  to  be  done  rather  than  'who'  does  it, 
or  'how'  it  is  done.  They  can  be  used  to  identify  and  organise  the  activities  that  a 
component  of  an  organisation  currently  performs,  or  should  be  performing  to  achieve  its 
objectives  and  goals  independent  of  the  organisational  structure  (Lonsdale  2000a, 
Lonsdale  2000b  and  Universitat  Bremen  2000b).  This  type  of  modelling  can  be  considered 
to  be  a  way  of  specifying  or  identifying  the  system  objectives,  but  not  to  compare  systems 
quantitatively. 

Use  cases  are  an  informal  and  imprecise  modelling  technique,  which  can  be  used  to  define 
the  fundamental  structure  of  an  application.  They  emphasise  describing  the  events  in  the 
story  of  interaction  between  actors  (external  agents,  which  normally  represent  the  roles  of 
people)  and  a  system  (usually  represents  a  software  system)  (Larman  1998  and  Lonsdale 
2000a  and  -b).  Use  case  modelling  cannot  usefully  be  used  to  capture  non-functional 
requirements.  Nor  can  it  be  used  to  capture  internal  functional  requirements.  The  use  case 
model  is  about  describing  'what'  the  system  will  do  at  a  high  level,  but  with  a  user  focus 
for  the  purpose  of  scoping  the  project  and  giving  the  application  some  structure.  Use  cases 
are  not  a  functional  decomposition  model  and  they  are  not  intended  to  capture  all  of  the 
system  requirements.  Use  cases  do  not  capture  'how'  the  system  will  do  anything,  nor  do 
they  capture  anything  the  actor  does  that  does  not  involve  the  system.  All  these  things  are 
better  modelled  using  other  techniques.  The  art  of  using  this  method  is  to  identify  the  user 
goals  not  the  system  functions.  Thus  it  is  not  a  good  technique  for  holistic  system 
evaluations. 
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Some  strategies,  such  as  systems  dynamic  modelling  (Coyle  1987;  Coyle  1996;  Kearney 
1998;  Coyle  1992;  and  Coyle  and  Millar  1996),  are  used  to  investigate  system  outputs, 
focusing  primarily  on  factors  associated  with  speed  of  operations.  This  type  of 
examination  focuses  on  the  dynamic  properties  of  the  system  and  investigates  its 
behaviour  over  time.  Its  strengths  are  in  its  representations  of  very  complex  systems  to 
provide  broad  indications.  Its  weaknesses  are  in  its  oversimplification  of  the  system's 
representation  and  reliance  on  time  as  its  major  measure.  It  is  best  suited  to  the 
investigation  of  strategic  outcomes  for  the  provision  of  first  estimates  to  focus  more 
detailed  investigations. 

Another  technique  having  a  time  and  process  focus  is  discrete  event  modelling  (Levis 
1993).  Generally  discrete  event  modelling  is  used  in  conjunction  with  the  functional 
decomposition  to  gain  a  model  of  a  particular  system.  Often  due  to  the  way  such  models 
are  constructed,  it  can  be  time  consuming  to  change  an  existing  model  to  represent  a  new 
instantiation,  particularly  if  unplanned  changes  occur.  This  technique  is  time,  process  and 
output  focused. 

Techniques  such  as  integrated  computer  aided  manufacturing  definition,  and  systems 
engineering  (US  DoD  2000a  and  -b)  are  designed  to  help  evaluate  systems,  but  from  a 
descriptive  viewpoint.  Also  they  are  very  focused  on  the  tangible  outputs  of  the  system  in 
terms  of  their  processes.  They  do  not  focus  on  the  operational  requirements  of  the  system 
as  a  whole.  Adjustments  using  these  techniques  are  labour  intensive.  These  techniques 
such  as  Integrated  DEFinition  language  are  a  structured  methodology  for  functional 
process  analysis.  They  suffer  from  the  complexity  of  the  diagrams  and  distinguishing 
between  the  'as  is'  system  and  the  'to  be'  system.  Other  problems  include  distinguishing 
between  the  controls  and  inputs  defined  for  the  methodology  and  establishing  proper 
boundaries  for  the  model. 

The  techniques  described  above  have  strengths  and  weaknesses,  some  of  which  are 
described  in  Richardson  and  Bartley  (1998).  Many  are  output  driven,  evaluating  the 
system  in  terms  of  the  quality  of  the  outputs,  allowing  limited  investigation  of  a  system  in 
terms  of  how  well  it  supports  the  production  of  these  outputs.  They  often  focus  on  using 
the  processes  to  define  and  describe  the  outputs  and  key  elements  of  the  system,  rather 
than  what  the  system  is  trying  to  achieve,  and  how  best  to  achieve  it.  Focus  also  tends  to 
be  on  certain  aspects  of  the  system  rather  than  the  system  as  a  whole.  The  techniques  often 
do  not  actually  evaluate  the  system  as  a  whole;  they  identify  different  aspects  of  it.  They 
have  the  ability  to  provide  descriptive  snapshots,  but  struggle  to  quantify  the  evaluation 
of  the  system  and  to  allow  for  comparison  of  system  instantiations. 

A  major  issue  with  analysing  complex  systems  is  the  fact  that  such  systems,  due  to  their 
complexity,  are  variable  in  nature.  This  means  that  even  for  a  given  set  of  environmental 
conditions  in  which  the  system  operates  there  are  many  possible  outputs.  So  a  complete 
analysis  with  any  output  based  techniques  is  very  expensive.  Thus  in  general  only  a  small 
fraction  of  the  possible  solutions  can  be  investigated.  Also,  process  based  approaches 
define  the  system  according  to  its  most  variable  nature;  the  processes  within  the  system.  It 
is  the  processes  that  change  constantly  and  are  very  dependent  on  the  human  elements  of 
the  system.  If  an  extensive  study  is  not  carried  out,  then  essentially  only  a  snapshot  of  the 
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system  will  be  obtained.  This  reduces  the  value  of  the  study  to  the  current  analysis  as  well 
as  the  value  it  has  to  future  evolutions.  Consequentially,  comparing  the  same  system 
under  different  circumstances  is  nearly  impossible  with  any  great  credibility  and 
comparing  similar  types  of  systems  can  be  even  more  problematic. 

What  is  needed  is  a  different  view  for  defining  the  system  and  focusing  evaluations;  this  is 
the  unique  nature  of  the  SIC  method.  Here  a  complex  system  is  defined  in  terms  of  how 
well  it  supports  the  achievement  of  the  outcome,  not  on  judging  the  quality  of  the  output 
of  the  system.  The  SIC  method  defines  a  generic  system  in  terms  of  its  variable  and  static 
properties.  The  variable  aspect  of  the  system  provides  a  foundation  for  comparative 
evaluations.  It  also  provides  a  scaling  ability  for  application  of  the  method,  if  required; 
more  detail  can  be  included  in  subsequent  studies.  Finally  it  is  independent  of  scenario, 
something  that  many  methods  are  not.  The  techniques  described  above  do  not  readily 
provide  all  of  these  capabilities.  However,  many  of  them  do  provide  tools  to  assist  in 
looking  at  certain  aspects  of  the  system  and  can  be  used  in  conjunction  with  the  SIC 
method. 


3.  The  System  Instantiation  Comparison  Method 

The  SIC  method  was  developed  to  allow  complex  systems  to  be  clearly  defined  and 
quantitatively  compared,  in  terms  of  supporting  the  achievement  of  an  outcome,  and  its 
variable  and  static  properties.  The  core  idea  behind  the  SIC  method  is  to  provide  a 
viewpoint  of  a  complex  system  in  three  parts.  Two  of  these  parts  are  defined  by  the 
system's  operational  requirements3  and  are  fixed  for  each  system  instantiation.  The  third 
provides  the  capability  of  the  system  to  support  these  requirements.  Once  this  breakdown 
has  occurred,  a  system  measure  can  be  determined  for  different  system  instantiations.  As 
with  all  modelling  techniques,  this  method  depends  on  the  correct  representation  of  the 
system  being  analysed.  The  method  is  considered  independent  of  scenario.  The 
operational  requirement  sets  the  context  in  which  the  system  is  investigated.  However,  the 
tempo  with  which  the  system  is  required  to  operate  influences  the  evaluation  of  the 
system.  Examples  of  this  scenario  independence  are  shown  in  Section  3.4.  The  SIC  method 
allows  the  quantitative  comparisons  of  system  instantiations  independent  of  scenario. 

The  method  can  be  used  in  many  areas.  It  can  be  applied  to  issues  such  as: 

•  Providing  near  real  time  analysis  and  results,  and  a  mechanism  for  'what  if'  situations. 

•  Providing  a  mechanism  for  the  aggregation  of  estimates  from  subject  matter  advisers, 
from  observations,  or  from  general  knowledge,  especially  in  support  of  seminars  or 
discussion  groups. 


3  The  use  of  the  term  operational  requirement  in  this  document  should  not  be  confused  with  the  use 
of  this  term  in  military  operations.  The  usage  here  is  more  general,  as  will  be  seen  by  the  definition 
given  in  Section  3.1.1. 
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•  Guiding  the  analyst  to  the  appropriate  war  game  and  extrapolating  the  results  of  the 
war  game. 

•  The  refinement  and  prediction  aspects  of  the  Battlelab  process  (Bowley  and  Lovaszy 
1999,  and  Bowley  1999).  If  the  method  is  linked  to  the  refinement  issues,  then  the 
method  should  be  linked  to  sensitivity  analysis  (an  area  of  future  work)  to  draw  out 
key  factors  associated  with  the  problem  under  study. 

•  A  predictive  method  to  help  guide  the  next  stage  in  the  development  of  a  problem 
under  investigation.  This  can  narrow  the  scope  of  interrogation  for  the  problems.  In 
the  case  of  war  games  it  can  focus  on  the  area  for  further  investigation.  This  is 
especially  relevant  when  big  changes  are  experienced. 

The  method  is  flexible  in  its  application  in  that  it  can  be  used  in  a  variety  of  different 
circumstances  and  varying  degrees  of  detail  depending  on  the  application  required.  It 
allows  data  to  be  reused,  and  like  systems  to  be  compared.  Application  of  the  method 
includes:  providing  a  capability  to  view  and  define  the  system  from  a  variety  of  different 
perspectives,  and  identification  of  critical  aspects  of  the  system.  These  aid  the  user  in 
clearly  defining  the  system  and  solving  issues  for  a  particular  application.  The  method 
provides  a  powerful  tool  to  describe  and  define  a  system  for  investigation  and  then 
compare  different  instantiations  of  the  system. 

When  viewing  or  investigating  a  system  it  is  important  to  identify  what  influences  the 
system.  Data  to  support  defining  the  system  comes  from  a  variety  of  places.  In  order  to 
define  the  system  using  the  SIC  method  it  is  essential  to  do  background  research  in  terms 
of  becoming  familiar  with  the  specific  system  of  interest  and  issues  which  could  influence 
it  and  its  components.  This  can  be  done  through  interviews,  reading  documentation 
available  about  the  system,  and  in  some  cases  questionnaires  could  be  used  to  get  key 
attributes  regarding  the  system,  which  are  relevant  to  the  study. 

Three  examples  are  provided  in  the  following  sections,  illustrating  the  application  of  the 
method  in  highlighting  the  different  components  of  the  system: 

•  A  biological  system's  ability  to  provide  the  required  nutrients  for  its  survival, 

•  A  defence  force  considering  a  new  capability  purchase  in  the  form  of  a  new  tank,  and 

•  A  control  organisation  looking  at  updating  its  standard  operating  procedures  to 
become  more  efficient. 


3.1  The  System  Model 

The  representation  of  the  system  shown  in  Figure  1  forms  the  basis  of  the  comparison  of 
the  system  instantiations.  The  model  is  composed  of  three  central  parts:  the  critical 
component,  the  system  functions  and  the  system  enablers.  The  critical  component  and 
system  functions  are  determined  by  the  operational  requirements  of  the  system  and  are 
static  from  one  instantiation  to  the  next.  These  are  considered  as  the  frame  of  reference  for 
comparisons  between  instantiations.  The  system  enablers  are  the  part  of  the  system  that 
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varies  between  different  instantiations  and  define  how  the  system  functions  operate  on  the 
critical  components  to  achieve  the  operational  requirements.  Identifying  the  components 
and  their  definitions  may  be  an  iterative  process  and  is  critical  to  the  successful  application 
of  the  method  in  order  to  compare  system  instantiations.  More  detailed  definitions  of  each 
part  of  the  model  along  with  illustrative  examples  are  given  in  the  following  sections. 


Operational 

Requirement 


Figure  1:  The  System  Representation 

3.1.1  System's  Operational  Requirement 

This  method  begins  with  the  definition  of  the  operational  requirement  of  the  system.  Its 
selection  greatly  influences  the  definition  of  the  other  parts  of  the  model  as  it  defines  the 
underlying  purpose  of  the  system. 

The  system's  operational  requirement  defines  the  highest  level  goal  of  the  system  and  sets 
the  environment  within  which  the  system  operates.  It  is  generic  and  static  in  nature.  Thus 
it  is  considered  to  be  independent  of  any  particular  circumstance  in  which  the  system  is 
placed.  The  operational  requirement  is  governed  by  how  the  system  works  in  its 
environment;  of  interest  is  what  influence  there  is  on  the  system,  in  particular  the  tempo 
with  which  the  system  is  required  to  operate.  A  different  system  scenario  does  not 
necessarily  change  the  overall  aim  of  the  system,  nor  the  components  within  the  system. 
The  speed  at  which  the  system  is  required  to  react  may  affect  the  interaction  of  the 
components  of  the  system,  not  what  the  components  are.  To  determine  the  operational 
requirement  of  the  system  the  analyst  should  answer  the  question  'how  does  the  system  as 
a  whole  attempt  to  influence  the  external  environment4?'  A  possible  tool  for  determining 
the  system's  operational  requirement  is  the  use  of  case  tools  (Larman  1998,  and  Lonsdale 


4  External  environment  refers  to  everything  outside  of  the  system 
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2000b).  Interview  techniques  and  questionnaires  are  another  option,  as  described  in 
Anselm  and  Corbin  (1990)  and  Minichiello  et  al  (1991).  The  operational  requirement  is 
independent  of  scenario  and  defines  the  highest  level  goal  of  the  system  in  terms  of 
external  influences  on  the  system. 

Examples  of  the  selection  of  an  operational  requirement  are  as  follows: 

•  In  the  biological  system  considered  here,  the  operational  requirement  is  nutritional 
survival. 

•  For  the  tank  procurement  example,  the  operational  requirement  could  be  considered 
to  be  the  'contribution  of  a  capability  to  enhance  a  combined  arms  team  in  carrying  out 
manoeuvre  operations  in  the  littoral  environment'. 

•  The  organisation  example  would  have  an  operational  requirement  of  'providing 
advice  and  monitoring  and  controlling  its  subordinate  organisations'. 

3.1.2  Critical  Component  and  Critical  Characteristics 

The  critical  component  of  the  system,  represented  on  the  left  side  of  Figure  1,  is  the 
essential  'thing'  the  system  needs  to  achieve  its  operational  requirement.  It  is  static  and 
generic  in  nature  and  is  applicable  throughout  the  whole  system  in  the  context  of  the 
system's  operational  requirement.  Determination  of  the  critical  component  is  achieved  by 
answering  the  question  'What  is  it  that  the  whole  system  needs  to  meet  the  operational 
requirement?'  Possible  tools  for  determining  the  system's  critical  component  include  soft 
systems  methods  (Finegan  1994  and  2000).  Another  method  is  to  use  incident  tracing 
techniques  in  a  holistic  manner,  through  the  use  of  interview  techniques  (Minichiello  et  al 
1991). 

Examples  of  critical  components  are  as  follows: 

•  In  the  case  of  a  biological  system,  with  an  operational  requirement  of  nutritional 
survival,  the  critical  component  is  defined  as  the  blood,  as  blood  is  the  part  of  the 
system  that  provides  the  delivery  of  nutrients  and  removal  of  waste.  Blood  alone  does 
not  ensure  a  biological  system  will  reach  its  goal  of  surviving  in  the  greater 
environment.  However,  it  is  the  critical  thing  the  system  uses  to  ensure  its  nutritional 
survival. 

•  In  selecting  the  right  type  of  tank,  the  critical  component  is  'effect'.  This  is  what  the 
whole  system  needs  to  have  to  meet  its  operational  requirement.  The  system,  in  this 
case  the  tanks,  provides  the  contribution  of  'effect'  to  'enhance  a  combined  arms  team 
in  carrying  out  manoeuvre  operations  in  the  littoral  environment'. 

•  In  the  case  of  the  organisation,  the  critical  component  is  the  'information'  that  the 
organisation  needs  to  undertake  its  operational  requirement  of  providing  advice,  and 
monitoring  and  controlling  its  subordinate  organisations. 

Depending  on  the  system  in  question,  the  method  provides  the  ability  to  specify  the  key 
characteristics  associated  with  the  system's  critical  component,  referred  to  as  the  system's 
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critical  characteristics.  The  critical  characteristics5  represent  the  attributes  associated  with 
the  critical  component  in  the  operational  requirements.  The  function  decomposition 
techniques,  see  Universitat  Bremen  (2000b),  can  be  used  to  identify  these  characteristics. 
The  number  of  characteristics  and  those  chosen  depends  on  the  level  of  the  study  and 
what  aspects  are  most  important  to  the  study.  For  more  detailed  studies  there  will  be  more 
characteristics  listed  than  for  a  higher-level  study.  These  critical  characteristics  are  a  list  of 
the  attributes  that  further  refine  the  definition  of  the  critical  component  and  allow  a  more 
detailed  assessment  of  the  system. 

Examples  of  critical  characteristics  are  as  follows: 

•  Consider  once  more  a  model  of  a  biological  system.  In  this  system  blood  was  defined 
as  the  critical  component.  Now  the  key  characteristics  of  blood  at  the  highest  level 
would  be  red  blood  cells,  white  blood  cells,  proteins  and  water.  However,  a  more 
detailed  study  may  require  these  to  be  broken  down  further  so  that  white  blood  cells 
were  further  divided  into  neutrophils,  macrophages,  lymphocytes  and  monocytes. 

•  In  the  example  of  the  tank  procurement,  the  critical  characteristics  that  make  up  the 
'effect'  are  identified  as:  fire  power,  protection,  sustainment,  and  mobility.  These  are 
arrived  at  by  not  considering  the  outcome  of  the  'effect',  but  decomposing  what 
capability  is  required  to  be  achieved  to  meet  the  contribution  of  'effect'  to  'enhance  a 
combined  arms  team  in  carrying  out  manoeuvre  operations  in  the  littoral 
environment'. 

•  In  the  case  of  the  organisation  with  a  critical  component  of  'information',  the 
organisation  views  the  information  in  a  different  manner.  The  attributes  of  the 
information  are  'quality',  'timeliness'  and  'accuracy'. 

It  is  also  important  to  remember  that  these  characteristics  are  static  in  nature. 

While  some  systems  may  have  the  same  critical  component,  the  critical  characteristics  may 
be  different.  For  example,  consider  the  case  of  an  organisation  with  a  critical  component  of 
'information'  and  critical  characteristics  as  described  previously.  Depending  on  the 
operational  requirement,  the  same  critical  component  may  break  down  into  different 
critical  characteristics.  An  example  of  a  different  set  of  critical  characteristics  for 
'information'  could  be  'type',  'format',  'transfer  rate'.  The  critical  characteristics  depend  on 
the  operational  requirement. 

3.1.3  System  Functions 

The  way  in  which  the  system  utilises  its  critical  component  and /  or  the  critical 
characteristics  to  meet  its  operational  requirement  is  defined  by  the  system  functions, 
shown  on  the  right  side  of  Figure  1.  These  functions  represent  the  activities  undertaken  by 
the  system  to  achieve  its  operational  requirement.  They  are  static  and  generic  in  nature. 
These  may  be  specific  to  given  aspects  of  the  system.  The  functions  of  the  system  are 
determined  by  answering  the  question  'how  does  the  system  use  the  critical  component  to 


5  In  the  remainder  of  this  document  the  critical  characteristics  are  sometimes  also  simply  referred  to 
as  characteristics,  when  it  is  clear  what  this  means. 
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meet  its  operational  requirement?'  A  possible  tool  for  determining  the  system's  functions 
is  data  flow  modelling  (Universitat  Bremen  2000a). 

Examples  of  system  functions  are  as  follows: 

•  For  the  biological  system  being  considered,  the  system  functions  could  be  defined  as 
add  nutrients,  remove  nutrients,  add  waste  and  remove  waste.  This  sets  the  functions 
of  the  system  at  a  very  high  level.  The  critical  component  level  of  abstraction  required 
for  the  study  determines  the  detail  of  the  functions.  For  example,  in  a  more  detailed 
study,  the  function  of  add  waste  might  become  the  two  functions  add  carbon  dioxide 
and  add  urea. 

•  For  the  case  of  the  tank  procurement,  the  functions  of  the  system  are  identified  as 
suppression,  destruction,  neutralisation,  manoeuvre,  canalise,  and  survive.  These 
describe  the  interaction  between  the  'effect'  and  the  contribution  of  this  'effect'  to  the 
operational  requirement.  So  the  system  utilises  its  critical  component  of  'effect',  that  is, 
fire  power,  protection  sustainment  and  mobility,  to  meet  its  operational  requirement, 
'enhancing  a  combined  arms  team  in  carrying  out  manoeuvre  operations  in  the  littoral 
environment',  using  the  system  functions  of  suppression,  destruction,  neutralisation, 
manoeuvre,  canalise,  and  survive. 

•  Using  the  organisation  system  as  an  example,  a  more  detailed  breakdown  of  the 
functions  is: 

•  Assessment  -  consisting  of  gaining  an  understanding  of  the  current  situation 

•  Monitoring  -  sourcing,  storing,  filtering,  sharing  and  accessing  information 

•  Generation  -  develop  products 

As  can  be  seen  in  the  examples  above,  the  system  functions,  as  with  the  critical 
component,  can  be  broken  down  into  as  much  detail  as  is  required  for  the  study. 

3.1.4  System  Enablers 

The  system  enablers,  shown  at  the  bottom  centre  of  Figure  1,  represent  the  variable  or 
dynamic  parts  of  the  system  as  they  change  between  instantiations.  To  determine  the 
enablers  of  the  system,  the  analyst  should  answer  the  question  'how  does  the  system  use 
the  available  'equipment/ resources'  to  allow  the  functions  to  utilise  the  critical  component 
to  meet  the  system's  operational  requirement?'.  Knowledge-based  system  development 
(ARTI  2000),  structured  systems  analysis  and  design  methodology  (Uni.  of  Glamorgan 
2000  and  Universitat  Bremen  2000c),  activity  modelling  (Fonsdale  2000a  and  Universitat 
Bremen  2000b)  and  computer  aided  manufacturing  definition  (US  DoD  2000a  and  -b)  are 
tools  that  can  be  used  to  help  determine  the  system  enablers. 

An  example  of  the  selection  of  the  enablers  of  the  system  can  be  seen  by  considering  the 
case  of  the  biological  systems  removal  of  non-gas  waste  from  the  blood  via  the  kidney.  The 
form  that  the  kidney  takes  varies  from  one  instantiation  to  the  next.  For  example,  in  birds 
the  kidney  uses  less  water  to  extract  waste  than  in  a  fish.  So  these  two  biological  system 
instantiations  have  the  same  critical  component  and  functions,  but  different  system 
enablers. 
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It  is  recommended  that  the  system  enablers  be  further  divided  into  three  groups: 
resources,  structure  and  mechanisms.  In  essence  this  division  of  the  system  enablers  is 
arbitrary.  The  key  requirement  is  to  separate  the  system  costs  from  the  other  enablers. 
Costs  or  resources  are  not  included  in  the  evaluation  of  the  system,  as  they  do  not  directly 
change  how  well  a  given  system  instantiation  can  achieve  its  goals6.  It  seems  logical  to 
group  the  other  enablers  into  those  relating  to  the  physical  layout  of  the  system,  the 
structure,  and  those  relating  to  how  the  resources  are  used,  the  mechanisms.  This  reduces 
the  time  required  when  studying  different  system  instantiations  as  the  grouping  of  the 
system  enablers  provides  a  logical  metric  that  can  be  associated  with  the  change  of  the 
system.  The  system  structure  and  mechanisms  provide  the  systems  functions  with  their 
ability  to  interact  with  the  critical  component  and  thus  are  shown  in  Figure  1  as  the  link 
between  the  system  functions  and  critical  component. 

The  system  enablers  can  be  considered  in  as  little  or  as  much  detail  as  is  required  for  the 
evaluation.  The  number  of  enablers  can  also  vary.  They  can  be  further  sub-divided  to 
represent  more  detailed  properties.  These  still  need  to  be  combined  in  some  way  to 
represent  an  overall  estimate  for  each  of  the  system  enablers.  For  simplicity  only  high- 
level  estimates  of  the  enablers  will  be  considered  in  the  discussions  in  this  document. 

The  simple  analogy  is  used  that  the  mechanisms  are  the  tools  used  by  the  system,  the 
resources  are  what  the  tools  use,  and  the  structure  is  the  way  you  arrange  the  tools  to 
transfer  and  gain  access  to  the  critical  component.  Examples  in  the  following  sections 
highlight  particular  mechanisms,  resources  and  structure. 

3.1.4.1  Resources  or  System  Cost 

The  resources  define  the  cost  of  a  given  system  instantiation.  Some  examples  of  resources 
are;  physical  equipment,  training,  and  manpower.  The  resources  associated  with  a 
particular  modification  of  the  system  provide  the  system  cost. 

Examples  of  resources  are  as  follows: 

•  For  the  biological  system,  clearly  such  things  as  the  organs,  circulatory  system  and 
food  are  system  resources,  as  would  be  the  genetic  coding  required  to  use  these 
systems. 

•  In  considering  the  tanks;  personnel,  equipment,  such  as  ammunition,  weapon  system, 
detection  system,  tracking  system  and  self-protection  system  could  be  considered  to  be 
the  resources. 

•  For  the  organisation:  these  could  be  considered  to  be  the  population  of  the  system: 
communications  links,  computer  equipment,  software  programmers,  personnel,  and 
data  base  maintenance. 


6  This  technique  aims  to  populate  the  benefit  side  of  a  cost  benefits  analysis. 
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3. 1.4.2  Mechanisms 

The  mechanisms  define  the  system's  ability  to  manipulate  and  interpret  the  critical 
component.  Care  must  be  taken  in  defining  the  mechanisms  and  the  system  functions,  as 
these  can  sometimes  be  confused.  As  an  aid  to  distinguishing  between  functions  and 
mechanisms,  a  function  must  always  have  to  be  done,  it  is  generic  in  nature.  Conversely, 
the  enabler  mechanism  is  how  you  do  the  functions. 

Examples  of  mechanisms  of  a  system  are: 

•  A  mechanism  of  a  biological  system  is  the  distribution  of  blood,  that  is,  the  pumping 
of  the  heart.  Another  mechanism  may  be  the  way  osmosis  is  used  to  remove  and  add 
gasses  to  the  blood.  The  key  difference  is  that  functions  are  enduring  while 
mechanisms  can  change.  For  example  there  is  always  a  need  to  add  or  remove  gasses 
but  the  way  it  is  done  may  vary  between  instantiations. 

•  In  the  case  of  the  tank  procurement,  examples  of  'mechanisms'  would  be  training, 
doctrine,  command  and  control,  rules  of  engagement. 

•  In  the  organisation,  there  is  always  a  need  to  store  information,  thus  the  storing  of 
information  is  a  function.  However,  the  use  of  a  given  filing  system  is  a  mechanism  as 
this  is  one  way  of  achieving  the  function  of  storing  information.  Other  mechanisms 
include  templates,  formats,  data  base  management,  standard  operating  procedures 
and  references. 

3. 1.4. 3  Structure 

The  structure  is  the  way  the  resources  are  arranged  and  determines  the  ability  of  the 
system  to  transfer  and  gain  access  to  the  system  critical  component. 

Examples  of  the  selection  of  the  structure  of  the  system  are: 

•  For  the  biological  example,  this  would  be  the  way  the  organs  and  circulatory  systems 
are  structured  in  a  biological  system. 

•  The  tanks'  'structure'  includes  the  layout  of  the  tank  itself,  the  equipment  fits,  and  the 
distribution  of  the  armour. 

•  The  organisation  'structure'  could  include  the  manning  distribution  and  layout. 
3.1.5  Summary  of  Components 

Table  1  provides  a  summary  of  the  components  identified  in  the  method.  The  operational 
requirements  for  each  system  are: 

•  For  the  biological  system;  nutritional  survival. 

•  For  the  tank  procurement  system;  contribution  of  a  capability  to  enhance  a  combined 
arms  team  in  carrying  out  manoeuvre  operations  in  the  littoral  environment. 

•  For  the  organisation;  providing  advice,  and  monitoring  and  controlling  its  subordinate 
organisations. 
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Table  1:  Examples  of  the  components  of  The  System  Instantiation  Comparison  Method 


System 

Critical 

component 

Critical 

characteristics 

System  functions 

System  enablers 

Resources 

Mechanisms 

Structure 

Biological 

system 

blood 

red  blood  cells 
white  blood  cells 

-  neutrophils 

-  macrophages 

-  lymphocytes 

-  monocytes 
proteins 

remove  waste 

add  waste 

-  add  C02 

-  add  urea 
remove  nutrients 
add  nutrients 

organs 

genetic  coding 
food 

circulatory  system 

distribution  of 
blood 
osmosis 

arrangement  of 
organs 

circulatory  system 
system  structure 

Tank 

procurement 

effect 

fire  power 
protection 
sustainment 
mobility 

suppression 

destruction 

neutralisation 

manoeuvre 

canalise 

survivability 

personnel 
equipment 
weapons  system 
detection  system 
tracking  system 
protection  system 

command  and 
control 
rules  of 
engagement 
training 
doctrine 

layout  of  tank 
equipment  fits 
distribution  of 

armour 

Organisation 

information 

quality 

timelines 

accuracy 

assessment 

monitoring 

generation 

population  of  system 
communication  links 
computer  equipment 
software 
programmers 
personnel 
data  base 
maintenance 

templates 
formats 
data  base 
management 
system 

standard  operating 
procedures 
references 

layout 

manning 

distribution 

u> 


Figures  2  to  4  provide  a  graphic  summary  of  the  three  examples  given.  They  focus  on  the  definition  of  the  systems  and  the  components  that 
make  them  up.  Table  2  provides  a  summary  of  the  steps  and  tools  used  to  describe  the  system  used  for  the  SIC  method. 
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Figure  2:  SIC  Method:  System  Definition  for  the  Example  of  a  Biological  System 
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personnel 


resources 


it  the  tools 


equipment 


training 


Critical  Characteristics: 
"Represent  the  attributes  associated 
with  the  critical  component" 


Effect 


doctrine 


mechanisms  : 

“can  be  described 


structure 


protection 

sustainment 

mobility 

firepower 


command  and  control 


rules  of  engagement 

layout  of  the  tank 
equipment  fits 
distribution  of  armour 


The  contribution  of  a 
capability  to  enhance  a 
combined  arms  team  in 
carrying  out  manoeuvre 
operations  in  the  literal 
environment. " 


Operational 
Requirement : 

How  does  the  system  as  a  whole 
attempt  to  influence  the  external 
environment?" 


Example  of  an  Army 
Tank  Procurement 


Enablers  : 

Variable  part  of  the  system 
and  changes  between 
Instantiations" 


Critical  Component : 

"What  is  it  that  the  whole 
system  needs  to  meet  the 
operational  requirement?" 


System  Functions: 

"How  does  the  system  use  the 
critical  component  to  meet  its 
^  operational  requirement? 


Figure  3:  SIC  Method:  System  Definition  for  the  Example  of  an  Army  Tank  Procurement 


Figure  4:  SIC  Method:  System  Definition  for  the  Example  of  an  Organisation 


understanding  of  current 
situation 


System  Functions: 

"How  does  the  system  use  the 
critical  component  to  meet  Its 
operational  requirement? 


templates 

formats 

standard  operating  procedures 
references 


Enablers  : 

Variable  part  of  the 
system  and  changes 
between  Instantiate 
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Table  2:  Summary  of  Tools  for  System  Representation 


Procedure 

Steps 

Possible  Tools 

Identify  system 
components  in 
terms  of  their 
generic  and  static 
nature 

Identify  operational 
requirement 

Activity  modelling,  structured 
systems  analysis  and  design,  use 
case  modelling,  interview 
techniques 

Identify  critical 
component 

Soft  systems,  system  dynamics 
modelling,  interview  techniques 

Identify  critical 
characteristics 

Activity  modelling,  functional 
decomposition,  interview 
techniques 

Identify  system  functions 

Data  flow  modelling,  use  case, 
functional  decomposition, 
interview  techniques 

Determination  of 
variable  aspects 
of  the  system 

Determine  enablers 

Soft  Systems,  interview 
techniques 

Mechanisms 

Integrated  DEFinition 

Language,  interview  techniques 

Structure 

Knowledge  based  system 
development,  interview 
techniques 

Evaluation'1 

Determine  weights 

Decision  analysis  tools; 
analytical  hierarchy  process, 
multi  criteria  analysis 

Enabler  values 

Simulation,  modelling,  subject 
matter  experts,  war  games, 
observation,  interviews, 
operations  research,  decision 
support  tools. 

This  aspect  of  the  SIC  method  is  discussed  in  following  sections,  but  has  been  included  here  to 
provide  a  complete  summary  of  tools. 
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3.2  Using  the  Method  for  Evaluation 

Once  the  system  model  is  defined  the  aim  is  to  access  how  'well'  the  system  enablers  allow 
the  system  functions  to  utilise  the  critical  component  (characteristics)  to  achieve  the 
operational  requirement.  The  value  that  results  from  this  assessment  is  referred  to  as  the 
system  measure. 

The  importance  of  a  given  critical  characteristic  may  vary  from  function  to  function.  We 
define  this  variation  as  the  criticality  index,  which  can  be  expressed  in  the  form  of  weights. 
Examples  of  techniques  for  determining  these  weights  include  decision  analysis  tools 
(Goodwin  and  Wright  1952),  such  as  decision  trees  (Marshall  1995),  analytical  hierarchy 
process  (Goodwin  and  Wright  1952;  and  Saaty  1980),  simple  multi-attribute  rating 
technique  (Goodwin  and  Wright  1952),  multi-attribute  utility  theory  (Kadvany  2000),  the 
data  envelopment  analysis  method  (Filar,  Gaertner  and  Lu  Wu  in  publication)  and  force 
field  analysis  (Vignaux  2000).  These  weights  are  fixed  and  do  not  vary  between 
instantiations  of  the  system;  they  are  determined  by  the  characteristics  and  functions, 
which  are  static.  In  the  case  of  the  biological  system,  a  given  characteristic  of  the  blood 
may  be  more  important  to  a  given  function,  for  example,  in  the  function  of  add  nutrient, 
red  blood  cells  are  of  great  importance  as  they  hold  oxygen,  while  white  blood  cells  are  of 
little  importance  as  they  do  not  carry  any  nutrients.  Note  that  these  weights  can  be  used  to 
calculate  the  overall  system  criticality  of  a  characteristic.  This  value  can  be  useful  in 
determining  which  characteristic  is  most  important  to  the  system. 

The  importance  of  a  system  function  to  the  overall  system  may  also  vary.  Thus  each 
function  is  also  weighted.  For  the  organisation  example,  the  tempo  of  operation  impacts 
on  the  system  functions  in  terms  of  their  weights. 

An  enabler  measure  is  determined  for  each  function  in  relation  to  each  critical 
characteristic  for  a  given  system  instantiation.  The  way  of  determining  this  measure  will 
be  system  dependent  and  some  possible  techniques  may  include  modelling  (Coyle  1996; 
Bowden  and  Pearce  2000;  Bowden,  Gabrisch  and  Davies  1997;  and  Fevis  1993),  operations 
research  techniques  (Taha  1992;  Gillett  1976;  and  Hiller  and  Fieberman  1973),  simulation 
and  war  gaming  (Bowley  and  Fovaszy  1999),  battle  lab  process  (Bowley  1999), 
observations  (Mills  and  Stothard  2000;  and  Rees  and  Kempt  in  publication),  surveys  (de 
Vaus  1995;  and  ABS 1993),  interviews  and  qualitative  techniques  (Strauss  and  Corbin  1990; 
and  Minchiello  et  al  1991),  and  decision  support  tools  (Uni.  Cambridge  2000).  These 
measures  are  then  combined  taking  into  account  the  criticality  of  the  system's  critical 
characteristics  for  the  given  function  and  the  importance  of  the  function.  This  process  is 
repeated  for  each  instantiation.  Once  the  initial  study  has  been  conducted,  subsequent 
studies  are  easier,  as  the  basic  definition  of  the  critical  characteristics  and  system  functions 
have  been  defined,  and  often  the  system  enablers  are  also  largely  defined,  reducing  the 
initial  overhead.  Also,  as  part  of  the  analyses,  the  system  costs  for  different  instantiations 
can  also  be  determined  from  the  resource  system  enablers. 
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The  SIC  method  does  not  make  any  judgement  on  the  quality  of  the  system's  outcome, 
only  an  the  evaluation  of  the  system's  ability  to  support  the  generation  of  that  outcome.  In 
this  way,  a  measure  for  evaluating  the  system  can  be  obtained  for  a  particular 
modification  in  terms  of  its  system  enablers.  The  key  elements  of  a  particular  system  are 
unique  to  that  type  of  system. 

3.3  Getting  One  Measure  of  the  System 

Table  3  provides  a  guide  for  collating  and  aggregating  the  data  required  to  support  the 
SIC  method.  In  Table  3  values  are  given  for  each  critical  characteristic  and  system  function 
pair.  For  function  FI  these  are  listed  in  the  table  as  the  values  Ai  and  Bi,  where  i  =  1  to  3. 
Two  values  were  used  in  this  sample  table  to  relate  the  two  types  of  system  enablers,  SE1 
and  SE2,  which  impact  on  the  system  measure;  structure  and  mechanisms.  Flowever,  the 
number  of  measures  is  up  to  the  analyst  and  this  may  vary  from  one  to  any  number 
depending  largely  on  the  type  and  scope  of  the  study.  This  information  is  the  raw  data 
that  is  used  to  determine  the  system  measure.  It  is  these  numbers  that  will  vary  between 
instantiations.  The  value  Al  in  Table  3  relates  to  how  'well'  the  instantiation  currently 
being  evaluated  allows  function  1  to  utilise  characteristic  1.  In  Table  3,  two  functions  have 
been  listed  so  values  for  the  second  function  are  also  entered  into  the  table;  values  Ci  and 
Di,  where  i  =  1  to  3.  Table  3  shows  how  all  three  variables;  functions,  critical  characteristics 
and  enablers,  can  be  represented. 

The  characteristic  weights  (weight)  are  also  given  in  the  table.  As  stated  earlier,  these 
determine  the  criticality  of  the  characteristics.  The  characteristic  weights  are  given  in  the 
table  as  WFjCk,  where  j  =  1  to  2  and  k  =  1  to  3.  Thus  there  is  one  weight  for  each  critical 
characteristic /  function  pairing.  These  weights  are  used  along  with  the  data  values  Ai,  Bi, 
Ci  and  Di,  i  =  1  to  3,  to  determine  the  function  characteristic  measure.  The  way  of 
combining  these  values  is  up  to  the  analyst7.  A  very  simple  linear  function  is  used  in  the 
headquarters  example  seen  in  Section  4,  but  any  applicable  function  can  be  used, 
Anderson  (2002);  Bridgman  (1922);  de  Neufville  (1990);  Fishburn  (1967);  Hwang  and  Yoon 
(1981);  Williams,  Bowden  and  Rees  (2001);  and  Yoon  and  Hwang  (1995)  provide  more 
examples  of  possible  ways  of  combining  values  to  form  an  overall  value.  The  outcome  of 
this  combination  is  given  in  Table  3  as  the  values  Mi,  where  i  =  1  to  6. 

The  importance  of  each  function,  the  function  weight,  is  given  in  the  table  as  Wfi  and  Wf2. 
These  values  are  used  to  get  an  overall  function  measure,  listed  in  the  table  as  FM1  and 
FM2.  The  user  is  required  to  determine  the  function  used  to  combine  the  function 
characteristic  measures  to  get  the  overall  function  measure. 


7  Some  possible  aggregation  functions  are  given  in  Hesser  (1991);  and  Williams,  Rees  and  Bowden 
(2000). 
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Table  3:  System  Measure  Determination 


Function,  Weight 

Function  1,  Wfi 

Function  2,  Wf2 

Weight 

SE1 

SE2 

Function  Characteristic 
Measure 

Weight 

SE1 

SE2 

Function  Characteristic 
Measure 

Characteristic  1 

Wfici 

A1 

B1 

Ml  =  F1(A1,  Bl,  Wfici) 

Wf2C1 

Cl 

D1 

M4  =  F4(C1,  Dl,  Wf2cd 

Characteristic  2 

Wf1C2 

A2 

B2 

M2=  F2(A2,  B2,  WFiC2) 

Wf2C2 

C2 

D2 

M5  =  F5(C2,  D2,  Wf2C2) 

Characteristic  3 

WfiC3 

A3 

B3 

M3=  F3(A3,  B3,  Wfics) 

Wf2C3 

C3 

D3 

M6  =  F6(C3,  D3,  Wf2q 

Function  Measure 

FM1=  G1(M1,  M2,  M3,  Wfi) 

FM2=  G2(M4,  M5,  M6,  WF2) 

System  Measure 

SM  =  H(FM1,  FM2) 
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The  final  manipulation  relates  to  the  calculation  of  the  system  measure  (SM)  in  Table  3. 
This  is  achieved  by  combing  the  function  measures8.  Again  the  function  used  for  this 
combination  depends  on  the  system  being  analysed.  One  issue  that  is  hidden  in  this 
analysis  is  that  of  emergent  behaviour  and  synergistic  effects.  These  are  reflected  in  two 
ways.  First,  in  general  because  we  are  looking  at  the  ability  to  carry  out  the  required 
functions  on  the  critical  component  using  the  system  outcomes  and  not  the  output  of  the 
system  these  effects  are  already  being  considered.  Second,  the  functions  used  to  calculate 
the  function  characteristic  measure,  overall  function  measure  and  the  overall  system 
measure  can  be  designed  to  allow  for  the  impact  of  these  issues.  These  functions  can  also 
be  designed  to  ensure  that  the  relationships  between  different  system  functions  and 
critical  characteristics  are  interdependent. 

3.4  Relationship  of  Analysis  to  Context 

One  of  the  problems  with  many  analysis  techniques  is  that  they  are  scenario  dependent. 
This  means  that  many  evaluations  need  to  be  conducted  to  determine  how  well  a  system 
performs  overall.  One  of  the  advantages  of  the  method  defined  here  is  that  the  measure  of 
the  system  is  determined  to  enable  it  to  have  a  broader  application  than  any  given 
scenario.  Thus,  a  measure  can  be  determined  for  a  broad  context9,  which  is  set  via  the  use 
of  the  operational  requirement.  This  is  because  the  measure  is  determined  from  the  ability 
of  the  system  to  support  the  outcome  and  is  not  focused  on  evaluating  the  quality  of  the 
outcome,  or  the  processes  followed  for  a  particular  output.  This  means  the  actual  activity 
in  which  the  functions  occur  is  less  important,  so  the  method  is  less  sensitive  to  the  specific 
scenarios. 

The  scenarios  are  accounted  for  in  terms  of  the  operational  requirements  of  the  system. 
The  evaluations  are  conducted  for  instantiations  within  the  same  operational  requirement. 
The  method  is  particularly  suited  to  investigating  the  introduction  of  new  technology  into 
a  system  and  assessing  its  impact  via  comparison. 

For  example,  in  the  case  of  the  biological  system  four  contexts  can  be  defined:  exerting 
activity,  normal  activity,  resting  and  sleeping.  Thus  the  context  in  this  case  depends  on  the 
nutritional  requirements  of  the  body  to  survive.  So  if  the  body  were  operating  under  stress 
there  would  be  the  need  to  supply  more  nutrients  to  the  cells  and  remove  more  waste  than 
when  it  is  at  rest.  For  the  analysis  of  the  biological  system  it  is  not  important  if  the  body  is 
walking  to  a  defined  objective,  just  whether  it  is  doing  this  under  the  contexts  of  exerting 
or  normal  activity. 

The  number  of  contexts  that  need  to  be  considered  depends  on  the  variability  in  the 
function  importance  between  each  context.  For  example,  in  some  biological  systems  it  may 


8  It  is  also  possible  to  combine  the  values  in  such  a  way  as  to  get  an  overall  characteristic  measure 
for  the  system  if  this  is  of  interest.  In  this,  the  function  characteristic  measures  are  combined  for 
each  characteristic. 

9  A  context  refers  to  a  set  of  like  scenarios  that  can  be  grouped  together. 
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be  necessary  to  define  another  context  relating  to  hibernation.  The  context  determines  the 
rate  at  which  the  operational  requirements  needs  are  to  be  met. 

The  critical  characteristic  measures  are  independent  of  context,  and  so  a  change  in  context 
simply  implies  a  change  in  function  importance.  Thus  to  determine  the  system  measure 
for  different  contexts  the  analyst  reassigns  the  weights  on  the  system  functions.  The 
measures  for  each  context  can  be  combined  using  methods  such  as  the  technique  for  order 
preference  by  similarity  to  the  ideal  solutions  method  (Filar,  Gaertner  and  Lu  Wu  in 
publication). 


4.  The  Military  Headquarters  Model 

The  primary  role  of  a  military  headquarters  is  to  provide  command  and  control  to  its 
subordinate  elements,  facilitated  through  the  use  of  information,  so  that  the  whole  military 
formation  can  operate  in  the  most  effective  manner.  As  discussed  in  Grinde  and  Hesser 
(1990),  information  is  a  driver  for  the  effective  operation  of  a  military  headquarters.  The 
commander  uses  information  to  make  decisions  that  control  the  forces  under  them  and  to 
disseminate  his/her  intent.  The  speed  of  making  a  decision  should  not  in  itself  be  the 
ultimate  goal  of  the  headquarters.  What  is  of  more  importance  is  the  'quality  and 
timeliness'  of  decisions.  Tied  up  in  this  issue  is  how  you  measure  the  quality  of  a  decision. 
It  is  not  sufficient  to  make  a  faster  decision  that  is  incorrect;  it  needs  to  be  a  quality 
decision  in  the  appropriate  timeframe.  Thus  a  study  of  timeliness  alone  does  not 
determine  how  well  the  overall  system  is  operating. 

To  manage  uncertainty  related  to  the  battlefield,  information  must  have  high  fidelity, 
currency,  consistency,  sufficiency  and  be  flexible  in  nature.  The  introduction  of  advanced 
information  management  systems  can  enhance  the  performance  of  command  and  control 
systems.  Central  to  this  performance  is  the  capability  to  provide  a  common,  coherent  and 
accurate  picture  of  the  environment  to  the  appropriate  command  elements.10 

The  way  in  which  a  military  headquarters  manages  and  utilises  its  information,  in 
particular  with  the  introduction  of  information  technology,  offers  the  potential  to  provide 
an  operational  edge  for  the  decision  processes  and  ultimately  the  whole  system.  It  is 
assumed  that  future  command  and  control  environments  will  be  drastically  changed  by 
the  introduction  of  new  technologies  and  the  information  age.  However,  these  technology 
advancements  by  themselves  do  not  necessarily  relate  to  an  increase  in  military 
effectiveness;  the  technology  must  be  employed  within  a  suitable  training  regime, 
organisational  structures  and  concepts  of  operations. 


10  These  concepts  are  an  extension  of  the  work  presented  in  Grinde  and  Hesser  (1990)  and  Nobel 
and  Wheatley  (1999). 
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There  is  a  requirement  to  determine  the  impact  of  the  introduction  of  new  technology  into 
a  military  headquarters  environment.  The  SIC  method  is  a  method  that  can  be  used  to 
achieve  this. 


4.1  Defining  the  SIC  Method  Components  as  Applied  to  a  Military 
Headquarters 

The  SIC  method  was  developed  through  a  requirement  to  provide  a  simple,  flexible  and 
cost-effective  approach  on  which  to  base  evaluations  of  command  and  control  systems. 
Therefore,  it  seems  appropriate  to  use  a  military  headquarters,  the  centre  for  command 
and  control,  as  a  typical  example  with  which  to  illustrate  its  application. 

A  military  headquarters  epitomises  the  concept  of  a  complex  pluralist  system,  as  defined 
in  Flood  and  Jackson  (1991).  Introducing  new  technologies  adds  further  complexity, 
resulting  in  changes  to  the  processes  and  cultural  aspects  of  the  headquarters.  It  therefore 
poses  a  challenge  for  evaluation  in  terms  of  generically  defining  the  headquarters  as  a 
system,  and  providing  a  flexible  and  cost-effective  method  to  conduct  comparisons 
between  instantiations. 

The  critical  component  of  a  military  headquarters  is  information.  In  understanding  the 
information  requirements  of  a  headquarters  it  is  necessary  to  break  the  critical  component 
into  critical  characteristics.  These  represent  the  key  information  requirements  to  perform 
the  headquarters'  functions.  The  critical  characteristics  are  the  generic  set  of  the 
commander's  critical  information  requirements.  As  supported  by  NATO  1998;  Australian 
Army  (1996);  Gridne  and  Hesser  (1990),  Hesser  (1991);  Lane  and  Galvin  (1999);  Lloyd 
Merfun  (1998);  Nobel  and  Wheatley  (1999)  and  collected  military  exercise  data,  they  are 
applicable  across  all  scenarios  and  can  be  extracted  from  the  standard  messages  received, 
transmitted  and  used  by  a  headquarters.  It  is  proposed  that  this  data  is  all  that  is  needed  to 
make  decisions  within  the  headquarters  and  all  data  that  flows  into,  out  of  and  within  the 
headquarters  fits  into  one  of  these  categories.  These  categories  are  generic  in  nature  and 
independent  of  level  of  command,  and  essentially  define  the  data  required  to  obtain 
situational  understanding  in  the  battlespace.  The  authors  have  determined  that  for  a 
military  headquarters,  critical  characteristics  are  (listed  in  no  particular  order): 

•  Own:  identity,  location,  status  and  intent 

•  Opposition:  identity,  location,  status  and  intent 

•  Neutral:  identity,  location,  status  and  intent 

•  Geospatial 

•  Environment  (including  weather) 

Key  activities  undertaken  on  the  information  by  the  system  to  achieve  its  operational 
outcomes  are  defined  as  the  system  functions.  The  focus  of  these  functions  is  towards 
monitoring,  understanding  and  trying  to  control  the  environment  the  headquarters  is 
immersed  in.  Once  the  environment  is  understood,  informed  assessments  can  be  made. 
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This  then  forms  the  generic  basis  for  defining  the  functions  associated  with  headquarters' 
operations11 .  These  functions  provide  the  required  information  to  the  decision  maker  and 
the  command  and  control  processes  that  enable  assessment,  planning  and  execution,  thus 
defining  the  system  functions. 

For  a  military  command  and  control  system,  such  as  a  military  headquarters12,  the  system 
enablers  associated  with  resources,  structure  and  mechanisms  provide  a  representation  of 
how  well  the  system  manages  the  information,  i.e.,  its  ability  to  interpret,  manipulate, 
access  and  communicate  information  internally  within  the  sub-system  and  between 
systems. 

Figure  5  provides  a  summary  of  the  components  associated  with  the  technique  for  the 
evaluation  of  a  military  headquarters. 


Figure  5:  Example  of  a  Military  Headquarters  Representation 


In  any  military  system  there  are  considered  to  be  three  different  tempos  of  operations: 
high,  medium  and  low.  These  define  the  demands  on  a  military  headquarters.  Thus  the 
three  contexts  for  a  military  system  are  defined  as  high,  medium  and  low  tempo. 


11  The  generic  nature  of  the  functions  of  a  command  and  control  system,  regardless  of  the 
technology  or  the  doctrine  is  supported  by  Grinde  and  Flesser  (1990)  and  Kirzl  (1999).  The  functions 
associated  with  command  and  control  are  discussed  further  in  Checkland  (1993);  Kearney  (1998); 
and  Bowden  and  Davies  (1998). 

12  While  both  the  example  of  organisation  discussed  in  Section  3.1,  and  the  military  headquarters, 
have  been  defined  as  having  the  same  critical  component,  due  to  their  different  operational 
requirements  they  have  different  critical  characteristics. 
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4.2  Application  of  the  Method  to  the  Evaluation  of  a  Military 
Headquarters 

To  show  how  the  method  works,  three  instantiations  of  a  system  are  compared.  The  first 
instantiation  is  considered  as  the  base  instantiation,  the  second  is  a  slight  enhancement  to 
one  component  of  the  system,  and  the  third  provides  an  enhancement  to  the  system  as  a 
whole.  What  will  be  investigated  is  the  impact  of  enhancing  the  information  on  friendly 
locations  in  a  headquarters  at  brigade  level.  In  the  second  instantiation  the  change  is  in  the 
form  of  regular  automated  updates  of  own  positions.  Instead  of  receiving  own  locations 
every  two  hours,  friendly  locations  will  be  automatically  sent  every  10  minutes  from  a 
global  positioning  system  attached  to  each  of  the  sub-units.  These  locations  are  then 
manually  added  to  the  battle  map.  In  the  third  instantiation  this  data  will  be  automatically 
placed  on  a  digital  battle  map.  Thus  the  overall  effect  of  both  instantiations  is  more  regular 
and  accurate  information  about  own  forces  locations. 

For  the  example  considered  here,  the  neutral  force,  identity  and  weather  information  are 
not  considered  to  be  significant  for  illustrative  purposes,  so  these  critical  characteristics  do 
not  need  to  be  included  in  the  analysis. 

For  the  changes  being  considered  in  the  example,  the  functions  planning,  execution  and 
assessment  are  assumed  to  be  the  most  affected.  This  will  help  to  simplify  the  explanation 
of  the  method.  Note  that  in  the  planning  and  assessment  functions,  the  information 
elements  are  transferred  and  accessed  internal  to  the  headquarters,  while  the  execution 
function  is  the  ability  to  access  and  transfer  information  external  to  the  headquarters. 
Representative  values  are  used  to  illustrate  the  method. 

The  variations  in  the  system  instantiations  considered  in  this  example  result  in  changes  in 
all  three  enablers;  resources,  structure  and  mechanisms.  For  example,  the  second 
instantiation  introduces  global  positioning  system  resources  to  lower  units  so  their 
position  information  can  be  passed  to  the  headquarters.  The  third  instantiation  introduces 
further  computing  power  into  the  headquarters  to  process  and  display  the  extra  location 
data. 

Table  4  provides  the  results  of  collating  and  analysing  the  data  for  the  first  instantiation 
required  to  support  the  method.  Values  are  given  for  each  of  the  information 
characteristics,  described  on  the  left  hand  side  of  the  table,  and  the  system  function,  given 
across  the  top  of  the  table.  For  each  function,  using  the  planning  function  as  an  example, 
two  values  are  used  to  relate  the  types  of  system  enablers,  El  (structure)  and  E2 
(mechanism),  which  impact  on  the  system  measure.  This  value  relates  to  how  'well'  the 
instantiation  currently  being  evaluated  allows  the  functions  to  utilise  a  specific 
information  characteristic.  It  is  these  numbers  that  vary  between  instantiations.  There  are 
many  ways  of  determining  these  values,  as  discussed  in  Section  3.2 

The  critical  characteristic  weights  are  also  given  in  the  table  as  weights.  These  determine 
the  criticality  of  the  characteristics  to  the  given  functions.  Thus  there  is  one  weight  for  each 
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critical  characteristic  function  pairing.  For  a  headquarters,  the  critical  characteristic 
weights  can  be  determined  using  a  criticality  index  as  done  in  Hesser  (1991).  This  allows 
for  the  calculation  of  the  weights  using  each  characteristic's  perishability,  frequency  and 
importance.  An  alternative  technique  would  be  to  use  the  analytic  hierarchy  process 
(Saaty  1980),  allowing  subject  matter  experts  to  rank  the  critical  characteristics  for  each 
function.  Yet  another  possibility  is  the  use  of  data  envelopment  analysis  (Boussofiane, 
Dyson  &  Thanassoulis  1991);  this  will  be  explored  further  in  Section  4.2.3. 


Table  4:  Evaluation  For  The  First  Instantiation 


FUNCTIONS 

Planning 

Assessment 

Execute 

Function 

Weight 

0.2 

0.6 

1 

Information 

Characteristics 

Weight 

El 

E2 

Weighted 

Function 

Measure 

Weight 

El 

E2 

Weighted 

Function 

Measure 

Weight 

El 

E2 

Weighted 

Function 

Measure 

Opposition 

Location 

0.8 

0.5 

0.5 

0.16 

1 

0.5 

0.5 

0.6 

0.8 

0.5 

0.5 

0.8 

Opposition 

Status 

0.5 

0.5 

0.5 

0.1 

0.6 

0.5 

0.5 

0.36 

0.7 

0.5 

0.5 

0.7 

Opposition 

Intent 

0.7 

0.5 

0.5 

0.14 

0.7 

0.5 

0.5 

0.42 

0.7 

0.5 

0.5 

0.7 

Own  Location 

0.8 

0.5 

0.5 

0.16 

0.9 

0.5 

0.5 

0.54 

1 

0.5 

0.5 

1 

Own  Status 

0.6 

0.5 

0.5 

0.12 

0.6 

0.5 

0.5 

0.36 

0.4 

0.5 

0.5 

0.4 

Own  Intent 

0.9 

0.5 

0.5 

0.18 

0.4 

0.5 

0.5 

0.24 

0.4 

0.5 

0.5 

0.4 

Geospatial 

0.8 

0.5 

0.5 

0.16 

0.3 

0.5 

0.5 

0.18 

0.8 

0.5 

0.5 

0.8 

Cross 

Characteristic 

Function 

Measure 

1.0 

2.7 

4.8 

System 

Measure 

8.5 

These  weights  are  used,  along  with  the  system  enabler  values,  to  determine  the  function 
characteristic  measure.  The  way  of  combining  these  values  is  up  to  the  analyst  and 
depends  on  the  system  being  studied.  A  simple  weighted  sum  is  used  in  this  example. 

The  final  element  of  the  evaluation  is  the  function  weights.  These  weights  determine  the 
criticality  of  the  functions  to  the  system.  Again,  the  analytic  hierarchy  process  could  be 
used  to  calculate  these  weights.  Another  technique  would  be  to  gather  data  relating  to  the 
length,  frequency  and  importance  of  each  function  to  the  headquarters. 

The  function  weights  and  the  function  characteristic  measures  are  then  combined  to 
provide  the  system  measure.  Again  the  function  used  to  combine  these  values  is  up  to  the 
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analyst  and  depends  on  the  system  being  studied.  Once  more  in  this  example  a  weighted 
sum  is  used.  Table  4  shows  the  evaluation  for  the  first  instantiation  (the  base  line  case). 

For  the  second  instantiation,  the  introduction  of  the  proposed  changes  in  the  system  can  be 
described  as  follows.  There  is  a  substantial  change  in  the  transfer  of  external  information 
into  and  out  of  the  headquarters.  This  increase  in  information  transfer  is  accompanied  by 
an  increase  in  communications  bandwidth  usage.  The  ability  to  access  this  information  is 
hampered  because  procedures  are  not  in  place  to  accommodate  such  a  large  volume  of 
information.  The  fusion  of  this  information  has  also  not  been  addressed  in  this 
instantiation.  Once  the  information  is  internal  to  the  headquarters,  its  transfer  is  also 
hindered  due  to  the  increased  volume.  The  impact  of  these  changes  is  reflected  in  the  bold 
values  in  Table  5. 


Table  5:  Evaluation  for  the  Second  Instantiation 


FUNCTIONS 

Planning 

Assessment 

Execute 

Function 

Weight 

0.2 

0.6 

1 

Information 

Characteristics 

Weight 

El 

E2 

Weighted 

Function 

Measure 

Weight 

El 

E2 

Weighted 

Function 

Measure 

Weight 

El 

E2 

Weighted 

Function 

Measure 

Opposition 

Location 

0.8 

0.4 

0.4 

0.128 

1 

0.6 

0.4 

0.6 

0.8 

0.5 

0.5 

0.8 

Opposition 

Status 

0.5 

0.5 

0.5 

0.1 

0.6 

0.5 

0.5 

0.36 

0.7 

0.5 

0.5 

0.7 

Opposition 

Intent 

0.7 

0.5 

0.5 

0.14 

0.7 

0.5 

0.5 

0.42 

0.7 

0.5 

0.5 

0.7 

Own  Location 

0.8 

0.3 

0.3 

0.096 

0.9 

0.4 

0.3 

0.378 

1 

0.4 

0.4 

0.8 

Own  Status 

0.6 

0.5 

0.5 

0.12 

0.6 

0.5 

0.5 

0.36 

0.4 

0.5 

0.5 

0.4 

Own  Intent 

0.9 

0.5 

0.5 

0.18 

0.4 

0.5 

0.5 

0.24 

0.4 

0.5 

0.5 

0.4 

Geospatial 

0.8 

0.5 

0.5 

0.16 

0.3 

0.5 

0.5 

0.18 

0.8 

0.5 

0.5 

0.8 

Cross 

Characteristic 

Function 

Measure 

0.9 

2.5 

4.6 

System 

Measure 

8.1 

The  changes  described  above  are  determined  using  the  most  appropriate  method 
available.  These  could  include  qualitative  and  quantitative  observation,  modelling  and 
simulation  of  parts  of  the  system  as  described  in  Section  3.2. 

For  the  third  instantiation,  the  introduction  of  the  proposed  changes  in  the  system  can  be 
described  as  follows.  There  is  a  large  change  in  the  transfer  of  external  information  into 
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and  out  of  the  headquarters.  This  increase  in  the  volume  of  information  transferred  is 
accompanied  by  an  increase  in  communications  bandwidth.  The  ability  for  the  whole 
system  to  manage  this  information  has  also  been  accommodated,  via  the  use  of  an 
automated  command  support  system  within  the  headquarters  to  facilitate  the  transfer, 
interrogation  and  manipulation  of  information  elements  associated  with  the  system. 
Table  6  shows  the  evaluation  of  the  3rd  instantiation.  Once  more,  the  values  that  vary  from 
the  baseline  are  in  bold. 


Table  6:  Evaluation  for  the  Third  Instantiation 


FUNCTIONS 

Planning 

Assessment 

Execute 

Function 

Weight 

0.2 

0.6 

1 

Information 

Characteristics 

Weight 

El 

E2 

Weighted 

Function 

Measure 

Weight 

El 

E2 

Weighted 

Function 

Measure 

Weight 

El 

E2 

Weighted 

Function 

Measure 

Opposition 

Location 

0.8 

0.6 

0.7 

0.21 

1 

0.5 

0.6 

0.66 

0.8 

0.8 

0.7 

1.20 

Opposition 

Status 

0.5 

0.5 

0.5 

0.10 

0.6 

0.5 

0.5 

0.36 

0.7 

0.5 

0.5 

0.70 

Opposition 

Intent 

0.7 

0.5 

0.5 

0.14 

0.7 

0.5 

0.5 

0.42 

0.7 

0.5 

0.5 

0.70 

Own  Location 

0.8 

0.8 

0.7 

0.24 

0.9 

0.7 

0.9 

0.86 

1 

0.9 

0.9 

1.80 

Own  Status 

0.6 

0.5 

0.5 

0.12 

0.6 

0.5 

0.5 

0.36 

0.4 

0.5 

0.5 

0.40 

Own  Intent 

0.9 

0.5 

0.5 

0.18 

0.4 

0.5 

0.5 

0.24 

0.4 

0.5 

0.5 

0.40 

Geospatial 

0.8 

0.5 

0.5 

0.16 

0.3 

0.5 

0.5 

0.18 

0.8 

0.5 

0.5 

0.80 

Cross 

Characteristic 

Function 

Measure 

1.1 

3.1 

6.0 

System 

Measure 

10.2 

Table  7  provides  the  overall  comparison  of  the  three  system  instantiations.  The  second 
column  gives  the  system  measure,  that  is,  the  measure  of  how  'well'  the  given 
instantiation  allows  the  system  to  perform  its  functions  on  the  information.  A  larger 
number  in  this  column  indicates  that  the  system  instantiation  is  better.  The  third  column 
gives  a  percentage  change  from  the  baseline  instantiation.  This  analysis  shows  that  just 
increasing  the  bandwidth  and  providing  own  location  more  regularly  does  not  increase 
the  effectiveness  of  the  system  as  a  whole.  However,  incorporating  this  change  into  the 
whole  system  via  the  management  of  the  information  elements  does  increase  the  system 
effectiveness. 
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Table  7:  Comparison  of  System  Instantiations 


Instantiation 

System  Measure 

Change 

1 

8.5 

- 

2 

8.1 

Insignificant 

3 

10.2 

Moderate 

This  analysis  defines  the  change  in  system  effectiveness  for  a  single  context.  The  next  step 
in  the  analysis  is  to  carry  out  an  analysis  of  the  system  over  all  defined  contexts. 

4.2.1  Ranking  the  Overall  System  Contexts 

Table  4,  Table  5  and  Table  6  illustrate  how  well  the  system  has  performed  through  the 
system  measure,  which  can  be  equated  to  how  effective  the  functions  are  performing  in 
the  headquarters  for  a  particular  instantiation.  In  order  to  rate  the  instantiations.  Table  8 
represents  a  more  detailed  study  incorporating  the  system  context. 

To  achieve  the  overall  system  evaluation  the  technique  for  order  preference  by  similarity 
to  ideal  solution  methodology  is  applied(Filar,  Gaertner  &  Lu  Wu  in  publication).  This 
effectively  ranks  the  overall  systems  in  order  to  maximise  them  against  the  headquarters 
functions.13 

It  is  important  to  note  that  the  examples  provided  are  shown  in  three  dimensions, 
however,  any  number  of  dimensions  can  be  adopted.  The  dimension  of  the  example 
would  depend  on  the  number  of  contexts  being  considered. 

In  the  technique  for  order  preference  by  similarity  to  ideal  solution,  the  main  criterion  for 
the  evaluation  needs  to  be  defined.  In  this  case  the  contexts  of  the  system  are  defined  by 
the  operational  tempo:  low,  medium  and  high  have  been  used  in  this  case14.  To  illustrate 
the  concept  it  is  assumed  that  the  data  calculated  so  far  has  been  for  medium  operations. 
Random  data  will  be  used  for  low  and  high  tempo  operations  and  for  instantiations  4  to  6 
for  medium  tempo  operations.  This  is  illustrated  in  Table  8.  The  system  measures,  given  in 
columns  two,  three  and  four  of  Table  8,  provide  a  summary  of  the  calculated  system 
measures  for  each  context.  Also  included  in  this  table  are  the  maximum  solution  and 
minimum  solution.  The  maximum  and  minimum  solutions  are  the  best  and  worst  possible 
instantiations.  In  this  case  they  are  taken  as  the  maximum  and  minimum  values  for  the  six 
instantiations,  however,  this  is  not  always  the  case.  The  maximum  and  minimum  values 


13  This  is  one  method  of  ranking  solutions.  An  alternative  is  presented  in  the  next  section  where  it  is 
used  to  rank  the  information  characteristics.  If  a  single  optimal  solution  is  to  be  found  then  game 
theory(Binmore  1992)  can  be  used.  However,  this  approach  is  not  of  interest  here  as  it  is  the  role  of 
DSTO  to  advise  on  the  value  of  solutions,  not  to  pick  one. 

14  The  number  of  contexts  is  analysis  dependent  and  will  be  influenced  by  the  sensitivity  of  the 
measures  to  the  different  contexts.  The  more  sensitive  the  measures,  the  greater  the  number  of 
contexts  required. 
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may  in  fact  be  an  instantiation  that  does  not  actually  exist.  For  example,  the  minimum 
value  may  be  based  on  the  minimum  acceptable  value  of  the  system. 


Table  8:  System  Instantiation  Measures  In  Relation  To  Context 


System 

Instantiation 

Low 

Tempo 

Medium 

Tempo 

High 

Tempo 

1 

7.61 

8.50 

9.18 

2 

9.41 

8.10 

7.78 

3 

9.33 

10.20 

10.66 

4 

8.91 

9.30 

11.26 

5 

9.27 

10.67 

11.49 

6 

9.74 

9.78 

10.17 

Maximum 

9.74 

10.67 

11.49 

Minimum 

7.61 

8.25 

7.78 

In  order  to  look  at  the  overall  operational  capability,  each  context  is  weighted  in 
accordance  with  the  importance  of  its  overall  capability.  For  this  example  it  is  assumed 
that  context  1  represents  a  low  tempo  of  operation  and  is  weighted  as  0.25,  context  2 
represents  a  medium  tempo  of  operation  and  is  weighted  as  0.5,  and  context  3,  the  high 
tempo,  is  weighted  as  1.0.  In  reality,  weighting  of  contexts  may  depend  on  factors  such  as 
the  likelihood  of  the  context  occurring  and  the  criticality  of  the  system  in  the  given 
contexts. 

Flaving  determined  the  weights  of  the  contexts,  the  instantiations  can  be  ranked  using  the 
technique  for  order  preference  by  similarity  to  ideal  solution  (Filar,  Gaertner  &  Lu  Wu  in 
publication).  This  determines  the  ratio  of  the  distance  that  a  given  instantiation  is  from  the 
maximum  solution  in  comparison  to  the  distance  it  is  from  the  minimum  solution.  The 
ratios  for  the  values  in  Table  8  are  given  in  Table  9.  The  closer  the  ratio  is  to  zero  the  closer 
the  system  is  to  the  maximum  solution.  Thus,  in  this  example  the  instantiation  that  is 
closest  to  the  ideal  is  instantiation  5  and  the  instantiation  closest  to  the  negative  ideal 
solution  is  instantiation  2. 


Table  9:  Instantiation  Ranking  in  Relation  to  Maximum  and  Minimum  Instantiations 


Instantiation 

Ratio 

1 

0.65 

2 

0.88 

3 

0.21 

4 

0.18 

5 

0.03 

6 

0.33 
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4.2.2  System  Cost 

Generally  the  next  step  considered  is  the  cost  of  each  system.  Due  to  the  way  the  system 
was  defined  this  is  given  by  system  resources.  Thus  for  the  above  example  the  changes  in 
cost  would  be  the  inclusion  of  such  resources  as  GPS  systems  for  lower  level  units  and 
changes  in  staff  due  to  automation  of  some  work  in  the  headquarters.  This  will  not  be 
considered  in  this  document,  but  is  the  focus  of  future  work.  Ultimately  the  SIC  method 
can  be  used  to  assist  in  a  cost-benefit  trade  off  analysis.  Thus  the  question  of  what 
instantiation  of  the  system  is  most  cost-effective  can  be  addressed. 

4.2.3  Calculation  of  Criticality  of  Information  Characteristics 

As  described  in  Section  3.3,  it  is  possible  to  determine  the  overall  system  criticality  for  each 
of  the  critical  characteristics.  This  data  provides  an  indication  of  the  relative  importance  of 
the  different  characteristics  of  information  to  the  headquarters  as  a  whole.  As  mentioned 
earlier,  there  are  numerous  ways  of  doing  this  including  the  analytical  hierarchy  process 
(Saaty  1980),  regression  equations  (Hesser  1991)  and  data  envelopment  analysis 
(Boussofiane,  Dyson  &  Thanassoulis  1991).  In  this  section  the  data  envelopment  analysis 
method  is  discussed  in  detail.  All  three  methods  are  discussed  in  more  detail  in  Williams, 
Rees  and  Bowden  (2000). 

Consider  the  use  of  two  measures  in  the  calculation  of  the  information  characteristic 
criticality15.  The  first  step  in  defining  the  characteristic  criticality  is  to  define  the  critical 
boundary.  The  standard  process  for  defining  the  critical  boundary,  is  based  on  linear 
programming  and  is  given  in  Boussofiane,  Dyson  and  Thanassoulis  (1991).  Next,  two 
lengths  are  calculated  for  each  information  characteristic.  The  first  of  these  is  the  vector 
length  from  the  origin  to  each  point,  defined  here  as  Dl.  Consider  now  the  vector  from  the 
origin  to  the  boundary,  which  passes  through  the  point  being  considered,  this  length  is  D2. 
The  criticality  index  is  then  the  ratio  of  the  original  vector  length  (Dl)  to  the  extended 
vector  length  (D2).  Thus  the  criticality  gives  a  ratio  reflecting  how  far  the  point  is  from  the 
critical  boundary. 

There  are  several  problems  with  using  linear  programming  to  define  the  optimal 
boundary.  The  use  of  linear  programming  to  define  the  optimal  boundary  means  that  it 
and  the  intersection  on  the  boundary  are  hard  to  calculate.  Also  there  is  a  problem  with 
the  ratios  generated.  That  is,  it  is  possible  to  get  points  on  or  very  near  the  boundary  that 
are  actually  far  from  optimal.  Ideally  the  only  solution  that  gives  a  ratio  of  one  should  be 
the  point  that  takes  the  maximum  possible  value  in  all  measures.  To  help  overcome  these 
issues,  instead  of  using  linear  programming  to  define  the  boundary  a  simple  circle  with 
following  equation  is  proposed  by  the  authors: 

y2  +  X2  =  (Xmax)2  +  (ymax)2, 

where  xmax  and  ymax  are  the  maximum  values  that  measures  x  and  y  can  take. 


15  Two  measures  have  been  used  for  clarity  of  explanation,  however,  the  formulae  for  n  measures  is 
given  later. 
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This  means  that  for  the  solution  (a,  b),  D1  is  given  by 

Dl(a,b)  -  Va2  +b2  . 

Also,  for  all  the  points,  D2  is  given  by 

D2(a,b)  =  ^{xmax  f  +  (ymax  )2  . 

So  the  ratio  will  be  given  by 

_  D\(a,b )  _  Vo2  +  b2 
D2(a,b )  jn.  V  +  fy 

V  V  max  /  v'  max  / 


Figure  6  shows  a  plot  of  the  ratio  when  the  boundary  is  defined  by  a  circle  and  where  xmax 
and  ymax  equal  four.  Figure  6  shows  that  there  is  a  favouring  of  solutions  near  the  axis.  For 
example,  the  solution  of  (4,  0)  is  considered  a  better  solution  than  (3,  2).  This  is  a  direct 
result  of  the  nature  of  the  Euclidean  distance  and  a  concave  hull.  In  fact  the  use  of  linear 
programming  actually  creates  a  tighter  boundary,  emphasising  this  characteristic  further. 
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As  an  alternative,  consider  a  boundary  defined  by  the  hyperbola 

X  XV 
y  _  ^  max  J  max 


(Eqn  1) 


In  this  case,  to  calculate  D2  we  must  first  determine  where  the  extension  of  the  point  (a,  b) 
intersects  with  the  hyperbola.  The  equation  of  the  line  going  through  the  origin  and  the 
point  (a,  b)  is  given  by 


b 

y=-x. 

a 


(Eqn  2) 


Solving  simultaneously  equation  (1)  and  equation  (2)  gives  the  intersect  point  as  being 

max  *  Y  max  T  IW  max  *  y  max  ) 


Thus 


D2(a,b )  = 


j  a  (*max  X  y  max  )  j  \  (*max  X  3W) 


(x  xv  ) 

\  max  J  max  / 


axb 


(a2  +b 2) 


Therefore, 

P  = 


V(a2+b2) 


(^max  X  y  max  )  ^2  j  b^) 

axb 


axb 


(^max  X  7  max  ) 


This  type  of  function  tends  to  favour  points  towards  the  centre,  as  shown  in  Figure  7  for 

Xmax  —  ymax  —  4. 
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A  third  case  is  considered.  It  is  this  alternative  that  is  currently  considered  to  be  the 
method  of  choice  for  determining  the  criticality  of  information  characteristics  and  more 
generally  for  determining  the  criticality  of  the  system  characteristics.  This  involves  the  use 
of  a  line  with  gradient  minus  one  and  going  through  the  point  (xmax,  ymax).  This  line  has  the 
equation 

y  +  x  =  Xmax+ymax-  (Eqn  3) 


The  intersection  of  equation  (2)  and  (3),  is  given  by 

^  max  y  max  )  b(x  max  y  max  ) 


a  +  b 


a  +  b 


Thus 


D2(a,b)  = 
which  gives 


Va2  +b2  (xmax+ymax) 


a  +  b 


a  +  b 


Xmax+yr 
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Figure  8  shows  that  this  boundary  gives  a  more  linear  solution  to  the  problem,  which 
favours  neither  the  edges  nor  the  centre. 


The  line  boundary  is  extended  to  allow  for  n  measures.  In  »-dimensions  this  boundary  has 
the  equation 


I 


i=l 


Consider  the  point  defined  by  A  =  (ai,  a2,  . . .,  an)  that  represents  the  measures  of  an 
information  type  where  n  measures  are  considered.  The  distance  from  the  origin  to  this 
point  is  given  by 


The  parameterised  representation  of  the  line  going  through  the  point  A  and  the  origin  is 
X(t)  =  (a1t,a2t,...,ant). 
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To  determine  where  this  line  intersects  the  optimal  boundary,  the  parameterised  line  is 
substituted  into  the  equation  of  the  boundary  giving 

n  n 

“h  f  —  X  max  i  • 

i=l  i=l 


So  the  value  of  t  is 

n 

Vx 

max  1 


t  = 


i=l 


z«. 

i=l 


This  gives  the  point  where  this  line  intersects  the  optimal  boundary  as 


1  = 


XXmaxi 


i=l 


i=l 


V  i=l 


i=l 


■Z’ 

i=l 


i=i  y 


So 


nr — Z! 

d2(a)= 


i=l 


therefore 


p=n 


i=l 


Vx 

maxi 


i=l 


The  actual  form  that  the  optimal  boundary  takes  depends  on  the  study.  This  choice  is  left 
to  the  analyst  but  above  examples  of  possible  boundaries  can  be  used  as  a  guide.  In  Yoon 
and  Hwang  (1995)  a  general  form  of  different  optimal  boundaries  is  considered. 

The  criticality  index  for  each  of  the  information  characteristics  can  be  achieved  by  using  its 
importance,  frequency  and  perishability  (Hesser  1991).  An  artificial  sample  set  of  data  is 
shown  with  a  rating  out  of  516  in  Table  10. 


16  So  maxx  =  maxy  =  5. 
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Table  10:  Perishability,  Frequency  and  Importance  for  Information  Characteristics 


Information 

Importance 

Perishability 

Frequency 

Characteristic 

Planning 

Monitoring 

Execution 

Planning 

Monitoring 

Execution 

Planning 

Monitoring 

Execution 

Friendly 

-position 

1 

2 

2 

2 

2 

4 

1 

3 

5 

-  status 

4 

4 

4 

3 

3 

3 

1 

1 

1 

-  intent 

5 

2 

3 

1 

1 

1 

1 

1 

1 

Enemy 
-  position 

3 

5 

5 

1 

1 

1 

1 

1 

1 

-  status 

3 

4 

4 

5 

5 

2 

1 

4 

4 

-  intent 

5 

7 

7 

3 

3 

1 

1 

1 

1 

Neutral 
-  position 

3 

4 

4 

5 

5 

2 

1 

4 

4 

-  status 

1 

2 

2 

2 

2 

2 

1 

1 

1 

-  intent 

1 

3 

2 

1 

1 

1 

1 

1 

1 

Topography 

4 

1 

1 

1 

1 

1 

1 

1 

1 

Weather 

2 

2 

2 

1 

1 

1 

1 

1 

1 

The  data  is  then  combined  using  the  data  envelopment  analysis  method  as  defined  above. 
This  gives  the  criticality  values  shown  in  Table  11  and  Figure  9.  This  was  achieved  using 
the  three  measures  of  perishability,  frequency  and  importance,  thus  n  =  3,  for  each  of  the 
functions  and  information  characteristics. 


Table  11:  Criticality  Value  for  Each  Information  Category 


Planning 

Monitoring 

Execution 

Friendly 
-  position 

0.16 

0.28 

0.44 

-  status 

0.32 

0.32 

0.32 

-  intent 

0.28 

0.16 

0.20 

Enemy 
-  position 

0.20 

0.28 

0.28 

-  status 

0.20 

0.24 

0.24 

-  intent 

0.36 

0.36 

0.28 

Neutral 

0.36 

0.52 

0.40 

-  position 

-  status 

0.16 

0.20 

0.20 

-  intent 

0.12 

0.20 

0.16 

Topography 

0.24 

0.12 

0.12 

Weather 

0.16 

0.16 

0.16 
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Figure  9:  Plot  of  Criticality  Values  Versus  Information  Characteristics 


5.  Discussion  and  Conclusions 


This  document  delineates  the  definition,  analysis,  and  evaluation  of  a  complex  system  by 
developing  a  method  that  allows  quantitative  comparative  estimates  of  system 
instantiations  to  be  made,  and  allows  for  the  calculation  of  criticality  values  associated 
with  the  main  components  of  the  system. 

The  SIC  method  models  the  system  as  three  parts:  system  critical  component,  system 
functions  and  system  enablers.  Two  of  these,  the  system  critical  component  and  system 
functions,  represent  the  system  operational  requirements  and  are  fixed  for  each  system 
instantiation.  The  system  enablers  provide  the  capability  of  the  system  to  support  these 
requirements  and  are  variable  in  nature.  Thus  in  comparing  different  system  instantiations 
it  is  only  the  enablers  that  change;  the  remainder  of  the  model  is  fixed.  Thus  the  system 
measure  is  the  dependent  variable  given  by  the  independent  variables,  namely,  system 
critical  component,  functions  and  enablers. 

In  terms  of  systems  of  systems  methodologies,  the  SIC  method  is  aimed  at  use  with 
complex  pluralist  systems,  although  it  is  also  applicable  to  both  complex  unitary  and 
complex  coercive  systems  (Flood  &  Jackson  1991).  The  advantage  of  this  method  is  the 
way  it  views  the  system  from  the  aspect  of  how  well  the  system  enablers  allow  the 
system's  functions  to  use  the  critical  components  to  achieve  its  operational  requirements. 
Other  methods  of  evaluation  focus  on  the  quality  of  the  outputs  of  the  system  or  on 
process  oriented  approaches. 
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The  SIC  method  does  not  make  any  judgement  about  the  quality  of  the  system's  outcome, 
only  an  the  evaluation  of  the  system's  ability  to  support  the  generation  of  that  outcome.  In 
this  way,  a  measure  for  evaluating  the  system  can  be  obtained  for  a  particular  modification 
in  terms  of  its  system  enablers.  The  key  elements  of  a  particular  system  are  unique  to  that 
type  of  system. 

The  SIC  method  is  especially  relevant  when  big  changes  are  experienced,  and  well  suited 
to  real  time  analysis  and  results  for  'what  if  situations';  providing  a  predictive  aspect  to 
help  narrow  and  scope  interrogation  of  problems.  This  method  has  the  advantages  of 
being: 

•  Simple; 

•  Flexible; 

•  Scalable; 

•  Defined  by  a  generic  representation  of  the  system; 

•  Widely  applicable;  and 

•  Scenario  independent. 

A  military  command  and  control  system  was  used  as  an  example  for  the  application  of  the 
method.  Generic  characteristics  associated  with  the  information  and  the  functions  used  by 
a  military  headquarters  were  used  as  the  basis  to  represent  the  components  of  the  system. 
System  measures  associated  with  different  instantiations  were  obtained  and  rated  using  a 
number  of  different  techniques.  This  analysis  incorporated  the  context  of  the  system 
through  the  system  functions.  The  criticality  of  the  information  for  headquarters 
operations  was  also  calculated,  and  ranked  using  the  development  of  a  new  approach, 
which  is  considered  useful  for  focussing  the  introduction  of  information  technologies.  This 
is  of  particular  importance  for  the  design  of  combat  support  systems  in  terms  of  issues 
related  to  information  degradation  and  priority  of  information  transfer  and  display. 

The  method  has  been  used  in  a  number  of  real  applications.  The  application  of  the  method 
was  particularly  beneficial  for  defining  the  system,  and  providing  a  focus  for  evaluation. 
Two  examples  are:  the  evaluation  of  intelligences,  surveillance  and  reconnaissance 
systems  in  East  Timor,  and  investigations  into  interoperability  between  American,  British, 
Canadian  and  Australian  armies  within  a  coalition  force. 

For  the  case  of  the  intelligence,  surveillance  and  reconnaissance  investigation,  information 
was  used  as  the  critical  component.  The  method  implemented  was  similar  to  the  examples 
of  the  military  headquarters.  New  technology  was  introduced  into  the  system  and 
instantiations  measures  were  taken.  The  method  also  allowed  a  holistic  view  of  the 
system. 

For  the  coalition  army  activity,  interoperability  was  used  as  the  critical  component  with 
factors  associated  with  interoperability  used  as  the  critical  characteristics.  In  this  case  the 
method  was  used  primarily  to  define  the  system  holistically  for  investigation. 
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There  are  two  areas  planned  for  future  work  on  the  SIC  method.  The  first  relates  to  the 
linking  of  the  outcomes  with  costs  to  allow  for  cost  benefit  trade-off  analysis.  The  other 
relates  to  the  consideration  of  parameter  sensitivity.  This  is  an  issue  that  has  not  been 
considered  in  this  document.  This  will  help  to  focus  the  data  gathering  activities  on  those 
parameters  with  greatest  sensitivity. 

The  SIC  method  is  demonstrated  as  providing  a  simple,  flexible  and  cost  effective 
approach  with  which  to  compare  complex  military  systems. 
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